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Preface 


This Final Report presents the findings from the Field-Focused Study, one of several 
components in the National Evaluation of the Comprehensive School Reform 
Demonstration (CSRD) Program. The U.S. Department of Education (ED) sponsored the 
study under a task order and contract administered by the U.S. Department of Health and 
Human Services’ Program Support Center (Task Order #6, Contract No. 282-98-0027), to 
COSMOS Corporation. The McKenzie Group, Inc. (TMG), served as a subcontractor to 
COSMOS. In this function, TMG conducted and completed a review of all CSRD-related 
studies supported by the U.S. Department of Education (COSMOS/TMG, 2001). TMG 
staff also comprised about one-third of the site visit teams. 


The Field-Focused Study and the entire task order were directed by Robert K. Yin, 
Ph.D., of COSMOS. Margaret K. Gwaltney, M.B.A., of COSMOS served as the deputy 
project director, and Dawn Kim served as project coordinator. Many other staff members 
from COSMOS and TMG collaborated in the study, which involved four rounds of site 
visits to 18 CSRD schools and the collection and analysis of student achievement data from 
these schools. Michelle LaPointe, Ph.D., of the Policy and Program Studies Service, U.S. 
Department of Education, served as the ED project officer for most of the evaluation 
(Kathryn Doherty, Ph.D., was ED’s first project officer). 


The report was mainly prepared by Robert Yin and Dawn Kim and is presented in two 
volumes. Volume I contains the main text. Volume II contains six appendices to Volume 
I, including short summaries of the 18 schools that were studied. The summaries, as well 
as data presented in Volume I, are based on 18 in-depth and detailed cases reviewed by 
each school and available separately from COSMOS (see www. cosmoscorp.com). 
Throughout, all schools are identified anonymously. 
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EXECUTIVE SUMMARY 


The Comprehensive School Reform Demonstration Program. Comprehensive 
school reform calls for a “whole-school” and coordinated approach to improve schools. 
The strategy differs from piecemeal and fragmented efforts that also in the past have 
seemed only to lead to short-lived changes. 


To stimulate whole-school reform across the country, Congress appropriated funds in 
FY 1998 for the U.S. Department of Education (ED) to start the Comprehensive School 
Reform Demonstration (CSRD) Program. ED allocated the funds on a formula basis to 
states, who made awards to support 1,840 mostly Title I schools “in need of substantially 
improving” their performance. Subsequent rounds of annual awards to support additional 
schools have continued through FY2003. 


In applying for and accepting these funds, schools were expected to implement 9 
components, one of which is an effective, research-based method or strategy. Together, the 
9 components comprise the comprehensive reform aimed at improving student 
achievement: 


CSRD Components 


No. | Component 


Effective, research-based methods and strategies 
Comprehensive design with aligned components 
Professional development 

Measurable goals and benchmarks 

Support within the school 

Parental and community involvement 

External technical support and assistance 
Evaluation strategies 

Coordination of resources 
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The awards are for three years and had to be a minimum of $50,000 per year. Schools 
can only receive a single CSRD award. Given these conditions, federal funds therefore are 
intended to serve as “seed money” for whole-school reform. Beyond this period of time, 
schools are to continue reform with their own resources. 
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The Field-Focused Study of CSRD. ED commissioned the Field-Focused Study of 
CSRD as one part of its national evaluation of the CSRD program. The study began by 
randomly selecting 18 schools from the first two annual rounds of CSRD awards (1998-99 
to 2000-01, and 1999-00 to 2001-02). Field data were collected through four site visits to 
each school, conducted during 2000-02. 


The objective of the Field-Focused Study was to provide initial feedback about the 
implementation of the CSRD program. The topics covered by the study and in this report 
include: 


1) The progress in implementing the 9 CSRD components by the 
18 schools; 


2) The role of district and state influences in implementing the program; 


3) The apparent pathways or strategies for implementing comprehensive 
school reform; and 


4) The early signs regarding the potential sustainability of the program at 
the school level. 


Conclusions on CSRD Implementation: A Mixed Pattern 


The main priority of the Field-Focused Study was to determine the extent and nature 
of CSRD implementation at the 18 schools. Overall, the schools’ implementation of CSRD 
at the 18 schools was uneven. Based on a 47-point instrument reflecting the 9 components 
and devised specifically for the Field-Focused Study, nine schools garnered 80 percent of 
the 47 points and could be labeled as “nearly fully” implementing CSRD. Another six 
schools garnered 65 percent of the 47 points and could be labeled as “partially” 
implementing CSRD. The remaining three schools all scored lower and were judged to be 
“poorly” implementing CSRD. 


Implementation of the 9 CSRD Components. Implementation also was uneven 
across the 9 components (to streamline the discussion, several related components have 
been paired): 


Component 1 (see Section 2.1 of the main report for more details): 
The bulk of the schools’ CSRD resources and attention has been 
devoted to the implementation of a research-based method. By 
the time of the final site visit, implementation of such a method 
was proceeding fully at 9 of the 18 schools, with partial or minimal 
implementation at the other 9; 
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Component 2 (See Section 2.2): 
Less than half of the 18 schools had reforms with a compre- 
hensive design, reflected by a comprehensive plan combined 
with staff awareness that CSRD should extend to a school’s 
entire way of doing business and all its operations—and not 
simply adding a new function or project or activity; 


Components 3 and 7 (see Section 2.3.1): 
Both professional development and external technical support 
and assistance had largely been devoted to the implementation 
of the research-based method, not necessarily comprehensive 
reform more broadly; 


Components 4 and 8 (see Section 2.3.2): 
Both the measurable goals and benchmarks and evaluation 
strategies had been devoted to tracking student performance, 
not necessarily implementation progress; 


Components 5 and 6 (see Section 2.3.3): 
High turnover among staff and students had resulted in transient 
levels of either support within the school or parental and com- 
unity involvement; 


Component 9 (see Section 2.3.4): 
Most schools were in a position to coordinate or converge re- 
sources, but resources for sustainability were still uncertain. 


District and State Influences. The Field-Focused Study also collected data about 
district and state actions potentially affecting the schools’ CSRD implementation. Some of 
these actions were part of the CSRD administrative procedures, because states implement 
CSRD by having districts apply competitively on behalf of some or all of their schools. In 
the process, both states and districts can support or monitor the schools’ CSRD efforts. 
The study found varying degrees of such support (see Section 3.1.1). 


More important than these procedures related directly to the administration of CSRD, 
the study uncovered other important state and district policies, not directly related to 
administering CSRD, that nevertheless influenced CSRD implementation (see Section 
3.1.2). Some conditions, such as extremely limited professional resources, had a negative 
influence on CSRD. Other conditions, such as the direct alignment of CSRD designs with 
district improvement plans and state standards, had an extremely positive influence. Other 
conditions reflected the ongoing dynamics of school systems—e.g., districts reducing 
financial support for all external research-based methods, a district allowing a CSRD 
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school to become a charter school, and a district deciding to merge two schools that 
happened to be CSRD schools. 


Strong district or state influence, creating a “vertical” alignment to the school level, 
led alternatively to either complementarity or conflict with CSRD. As examples on the 
complementary side, CSRD provided two schools with resources and a compatible reform 
agenda to respond to their designation as underperforming schools in the state’s 
accountability system. As examples on the conflicting side, the shifting content of state 
assessments led districts to use resources for alternative curricula and professional 
development that were contrary to those involved in a CSRD school’s original plans or 
implementation. In general, these external state and district conditions appear to be highly 
relevant to CSRD implementation. 


Conditions Associated with Successful Implementation and the Role of State and 
District Influences: Three Pathways to Reform 


The study identified three different sets of conditions, or “pathways,” that appeared to 
be associated with the successful implementation of CSRD (see Section 3.4). The first 
pathway is a component-driven pathway, whereby a school uses the 9 CSRD components 
to guide the development and implementation of a comprehensive reform. The second is a 
method-driven pathway, whereby the school adopts and implements a comprehensive 
research-based method that affects virtually all school operations and whose successful 
implementation substitutes for the need for any independent articulation of the 9 CSRD 
components. (However, many research-based methods focus on specific curricula and are 
not comprehensive.) The third is a vertical-driven pathway, whereby a school articulates 
and pursues the needed comprehensive strategies as a result of state and district 
requirements involving: the setting of standards, use of appropriate assessment tools, and 
required alignment of district- and school-based strategic planning and improvement plans 
to meet state performance standards. 


No single pathway was considered the “best” or preferred pathway, and no pathway 
was necessarily more immune than the others to such disruptive conditions as: high 
principal turnover rates, limited professional development resources, or planned or 
unplanned school restructuring. 


Sustainability of “Whole-School” Reform: Still Questionable Given Current Fiscal 
Climate 
As a final topic, the study examined the prospects for sustaining school reform beyond 


the final year of CSRD funding (see Section 3.5). 
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Neither the original legislation nor ED defined the exact nature of a school’s changes 
to be associated with sustaining a comprehensively reforming school beyond the three-year 
CSRD award period. As a result, the Field-Focused Study examined two different views of 
sustainability and judged the 18 schools according to both. 


The first view, based mainly on the experiences of the New American Schools 
initiative, holds that the central changes to be sustained should be the practices associated 
with the originally-supported research-based method. The second view is that 
comprehensive reform, though embracing a research-based method, also transcends it. By 
this second view, successful sustainability would not necessarily be associated with the 
continued use of any particular method but could involve transitions from one research- 
based method to another, over time. The transitions would have to reflect a progression 
toward continued school and student improvement rather than the “churning” of innovative 
practices. 


Using the most lenient benchmark and accepting either of these two views as a 
criterion for assessing sustainability, 14 of the 18 schools were exhibiting a promising level 
of sustainability by the time their CSRD awards were ending. Accepting only the second 
view reduces the number to 11, and accepting the first view reduces it to 10. 


The main barrier to sustainability continues to be the limited availability of sufficient 
resources, especially in light of states’ and districts’ revenue shortfalls in recent years. To 
sustain a reforming process, even when existing resources have been coordinated and 
targeted to reform, still requires discretionary funds to support such essential activities as: 
adequate professional development (including support for teacher substitutes), especially in 
situations of high teacher turnover; time for common planning periods or teachers’ work on 
school leadership teams; and support for external technical assistance. Though such needs 
can be served with modest levels of funds, serving the needs is still a discretionary activity 
that may have to be ignored if core school operations are underbudgeted. 
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1. THE COMPREHENSIVE SCHOOL REFORM 
DEMONSTRATION PROGRAM’ AND 
THE FIELD-FOCUSED STUDY 


1.1 COMPREHENSIVE SCHOOL REFORM 


The Comprehensive School Reform Demonstration Program (CSRD) was designed 
to stimulate schoolwide change. The Comprehensive School Reform Demonstration 
(CSRD) Program provided formula grants to states. The states in turn invited districts to 
compete for subgrants, on behalf of schools, to implement comprehensive school reform. 
The program’s goal was to improve student achievement and enable all children to “meet 
challenging state content standards and performance goals” through reform strategies (U.S. 
Congress, 1998).” 


Participating schools were expected to: 


e Implement CSRD’s nine components’ (see Exhibit 1-1), one of 
which was the implementation of an effective, research-based 
method or strategy;* 


e Stimulate schoolwide change covering virtually all aspects of 
school operations, rather than implement a piecemeal, 
fragmented approach to reform; and 


'The Comprehensive School Reform Demonstration (CSRD) Program is the name of the education 
reform grant program as it was created in 1998. The program was subsequently renamed the Comprehensive 
School Reform (CSR) Program in P.L. 107-110, the No Child Left Behind Act of 2001 signed by the President 
in January 2002. Because the Field-Focused Study was commissioned to study the first two cohorts of CSRD 
schools (1998-1999 and 1999-2000 grantees), this final report will refer to the program as CSRD. 


The program was initially established through appropriations language, not authorizing legislation 
(Appropriations Act for the U.S. Department of Education, P.L. 105-78). In subsequent years, this 
appropriations language continued to be used to authorize the program, until the enactment of P.L. 107-110. 


°The CSRD program called for the implementation of nine reform components (referred to throughout 
this report). The redesigned CSR program under P.L. 107-110 now calls for eleven components. The two 
new components cover “support to the school’s staff” and the use of a research-based method (based on 
comparison group designs or on other strong evidence of improved student achievement). 


“The original congressional legislation listed 17 specific methods as examples, although schools were 
not limited to adopting these methods. 
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Exhibit 1-1 


DESCRIPTION OF NINE COMPONENTS OF REFORM 
IN CONGRESSIONAL PROVISIONS FOR CSRD 


1. Effective, research-based methods and strategies: Employ innovative strategies and 
proven methods for student learning, teaching, and school management that are based on reliable 
research and effective practices, and have been replicated successfully in schools with diverse 
characteristics. 


2. Comprehensive design with aligned components: Have a comprehensive design for 
effective school functioning, including instruction, assessment, classroom management, 
professional development, parental involvement, and management—that aligns the school’s 
curriculum, technology, professional development, etc. into a schoolwide reform plan designed to 
enable all students [including children from low-income families, children with limited English 
proficiency, and children with disabilities] to meet challenging state content and performance 
standards and addresses needs identified through a school needs assessment. 


3. Professional development: Provide high-quality and continuous teacher and staff 
professional development and training. 


4. Measurable goals and benchmarks: Have measurable goals for student performance 
[tied to the State’s challenging content and student performance standards, as those standards are 
implemented] and benchmarks for meeting those goals. 


5. Support within the school: Are supported by school faculty, administrators, and staff. 


6. Parental and community involvement: Provide for the meaningful involvement of 
parents and the local community in planning and implementing school improvement activities. 


7. External technical support and assistance: Utilize high-quality external technical 
support and assistance from a comprehensive school reform entity (which may be a university) with 
experience or expertise in school-wide reform and improvement. 


8. Evaluation strategies: Include a plan for the evaluation of the implementation of school 
reform and the student results achieved. 


9. Coordination of resources: Identify how other resources (federal/state/local/private) 
available to the school will be utilized to coordinate services to support and sustain the school 
reform effort. 


“Conferees’ original language in House Report No. 105-309, accompanying P.L. 105-78. 


“Language in brackets was added to the conferees’ language for the original “Guidance on the 
Comprehensive School Reform Demonstration Program,” issued by the US Department of Education on 
March 13, 1998; the language is present in the subsequent update dated August 11, 2000. 
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e Assess the impact from implementing comprehensive 
school reform in terms of changes in student achievement. 


For most of the CSRD funds, the eligible K-12 schools were Title I eligible schools, 
particularly those schools in need of substantially improving their student achievement 
levels. A small portion of the funds was available to all schools in need of such 
improvement. As a result, Title I schools represented the great bulk (over 85 percent) of 
CSRD schools. 


CSRD funding was disbursed to nearly 4,000 schools by FY 2001. CSRD funds first 
became available in FY1998, when $145 million in federal funds were allocated to state 
education agencies (SEAs).° The SEAs then made competitive three-year subgrants to 
their districts, eventually supporting 1,840 schools with the first year’s funds.° The awards 
(a minimum of $50,000 per year) were intended to stimulate or provide financial incentives 
for reform, but not support the entirety of reform.’ 


In FY1999, $145 million in funds were again allocated to the states, supporting the 
continuation of the initial awards and also the startups of a small number of new awards. In 
FY2000, the program awarded $220 million to the states, anticipating the continuation of 
the original schools into their third year but also providing the opportunity to make awards 
to a new set of schools. Similarly, $260 million was provided in FY2001, which permitted 
the funding of an entirely new set of schools starting in 2001-02, as well as the 
continuation of those schools begun with the FY2000 increase. The total number of 
schools funded under the CSRD program reached 3,875 by the 2001—2002 academic year 
(SEDL, 2002). For 2002-03, the appropriation increased funding for the newly defined 


Funds appropriated under the Title I program are allocated to States based on each State’s share of Title I 
Basic Grants in the previous fiscal year and may be used to support schools eligible to receive funds under Part 
A of Title I. Funds under FIE are allocated to States based on each State’s share of school-aged children, and 
may be used to support any school, including those not eligible to receive funds under Part A of Title I. Under 
both the Title I and FIE CSRD programs, ED was able to reserve up to 1 percent of the total amount 
appropriated for grants to outlying areas and the BIA, and up to 1 percent for national evaluation activities. 


°A study of 40 responding states (representing $103 million of the original $145 million in federal 
funds awards in FY1998) revealed that 3,001 school applications had been received from 1,088 school 
districts and that the average CSRD award had been $68,000 per school per year (New American Schools, 
1999). 


’The states do not necessarily make all of their subawards in the same academic year, and states also 
are permitted to set aside up to five percent of the funds for state administration, evaluation, and technical 
assistance. States also have the option to set aside award money for use by districts, and this may or may not 
be reflected in calculations of the average school award. 
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CSR program to $310 million, enough to support grant awards to an estimated 2,000 
additional schools (see Exhibit 1-2). 


CSRD called for research-based methods and strategies. The CSRD program drew 
its rationale and design from a longstanding and broad research and policy literature.® For 
instance, from a research perspective, two contemporary analysts (Desimone, 2000, p. 36; 
and Doherty, 2000, p. 3) noted that the theoretical underpinnings for comprehensive school 
reform were similar to and extended as far back as research on effective schools in the 
1970s and 1980s (e.g., Edmonds, 1979; and Cohen, 1982), among other educational 
strands. 


From a policy perspective, the practical underpinnings for CSRD emanated from the 
events leading to the redesign of Title I of the Elementary and Secondary School Act as 
amended by the Improving America’s Schools Act (IASA) of 1994, which shifted the 
federal government’s compensatory education investments away from supplemental and 
remedial (pull-out) programs toward standards-based, whole school reform—as reflected 
by a renewed appreciation for schoolwide programs and local flexibility (Shelley, 1997; 
Berends and Kirby, 1999; Puma and Drury, 2000; and Borman, 2000). As one result of the 
policy shift, the number of schoolwide programs within Title I increased from 1,200 in 
1991 to 9,000 in 1997-98 and represented 50 percent of all Title I schools (Desimone, 
2000). 


Coinciding with these research and policy developments, a critical catalytic influence 
was the ongoing experience of the New American Schools (NAS) initiative (e.g., Bodilly, 
1998; Berends, Bodilly, and Kirby, 2002). The New American Schools Corporation was 
formed in 1991 to design innovative, ‘break-the-mold’ schools. NAS’s initial wave 
supported eight research-based methods," which were then promoted by over 1,500 
schools in 11 partner communities (New American Schools, 1999). The initiative came at 
a time when the supply of research-based methods converged with the demand for 
continued school reform among low-performing schools (Bodilly, 2001). Early lessons 
from the initiative included a shift away from an emphasis on method development-design- 
only to include external technical assistance and support to schools (NAS, 1999), as well as 


®The present report, however, is not the occasion for a comprehensive review of this literature. 


°Throughout these policy transitions, an important undercurrent continued to be that “teachers matter” 
(Puma and Drury, 2000). Operationally, the assumption has been that changed vision and school-based 
planning will lead to changed classroom practices. All these practices will then produce the desired change 
(and improvement) in student performance. That “teachers matter” therefore suggests that policymakers need 
to focus on classrooms—the “operational core of education systems” (Puma and Drury, 2000, p. 33). 


‘Six of these eight methods were listed among the CSRD legislation’s original 17 methods. For a 
crosswalk between both groups and the top 25 methods or practices adopted by the first cohort of CSRD 
schools, see COSMOS, May 2000, Exhibit 1. 


1-4 


Federal Fiscal Year 


Exhibit 1-2 


SUMMARY OF CSRD AWARDS 


Amount Allocated | Cohort: Number of Schools* 


School Year First 


Appropriation (In Millions) Implemented 
1998 $145" Cohort 1: 1,840 1998-1999 
1999 $145" Cohort 1: 1,840 1999-2000 
Cohort 2: 146 
2000 $220° Cohort 1: 1,856 2000-2001 
Cohort 2: 146 
Cohort 3: 796 
2001 $260° Cohort 2: 146 2001-2002 
Cohort 3: 796 
Cohort 4: 1093 
2002 $310° Cohort 3: 796 2002-2003 
(No Child Left Behind) Cohort 4: 1093 
Cohort 5: 2,000 (est.) 


“ Amounts reported by States for each fiscal year do not always add to the amount allocated to States in 
that fiscal year (do to inconsistencies in State reporting). 

> Includes $25 million provided under the Fund for the Improvement of Education (FIE). 

* Includes $50 million provided under the Fund for the Improvement of Education (FIE). 

“ These are the appropriated funds (including $75 million in FIE funds) and the estimated number of 
schools the funds will support. 


Sources: Comprehensive School Reform Program: State Allocations for all Fiscal Years, U.S. Department of 
Education, June 25, 2002, www.ed.gov/offices/OESE/compreform/allyears.html. 

CSR Awards Database, Southwest Educational Development Laboratory, August 2002 (Note: The 
number of schools receiving awards was taken from the SEDL database, which was undergoing 
revisions at the time of this final report; the final number of schools awarded funds may have 
changed). 


an increased appreciation for the role of the district in supporting or facilitating 
comprehensive school reform (NAS, 1999). 


Emerging studies of comprehensive school reform have shown mixed outcomes. 
Some early studies suggested that the outcomes can be promising, as reflected by syntheses 
of these studies (Desimone, 2000; and Datnow and Stringfield, 2000). However, other 
studies have found no significant effects of comprehensive school reform on student 
achievement (e.g., Berends et al., 2002). The completed research includes a variety of 
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evaluations of specific research-based methods (e.g., Slavin and Madden, 2000),"* analyses 
of student achievement outcomes in entire school systems (e.g., Ross et al., 1999), and 
policy studies of strategies for educating disadvantaged children (e.g., Stringfield et al., 
1997). In addition, the U.S. Department of Education funded 46 studies, '* whose progress 
was tracked and documented to encourage coordination among the studies (COSMOS/ 
TMG, 2001), including the several components of the National Evaluation of CSRD. Yet 
other studies and related activities are being tracked by the National Clearinghouse for 
Comprehensive School Reform (NCCSR). 


Earlier research also includes individual case studies of CSRD-like comprehensive 
school reform, such as five case studies commissioned by the Consortium for Policy 
Research in Education—CPRE (Odden, Archibald, and Tychsen, 1999a and 1999b; 
Archibald and Odden, 2000; Odden and Archibald, 2000; and Fermanich, Odden, and 
Archibald, 2000). Another multiple-case study covered nine elementary schools engaged 
in comprehensive efforts similar to the CSRD program (The Dana Center, 1999). Among 
other findings, the cross-case analysis suggested that reform was more likely to succeed 
when efforts focused on boosting the schools’ instructional leadership (e.g., the capacity of 
principals and the channeling of resources, including technical assistance, to enhance 
instructional leadership), and when these efforts were combined with clear, measurable, 
and rigorous school accountability provisions. 


Overall, the available and ongoing research demonstrates that the CSRD program 
emerged against a solid backdrop of prior conceptual work and empirical evidence. At the 
same time, the experiences have raised at least two ongoing issues worth mentioning.” 
First, “CSRD” is not synonymous with “comprehensive school reform.” As used in the 
present report, CSRD has been defined specifically to represent the congressionally 
legislated program that started in 1998—its major distinction being the identification of 
nine specific components to be implemented as part of comprehensive reform. In contrast, 


"One criticism is that very few of the research-based methods “...have been the subject of high quality 
evaluations that justify confidence in their effectiveness,” and that in addition, “...the few models that do have 
a substantial amount of research...are models for which evaluations were conducted mainly by the design 
team, not objective third-party evaluators” (Desimone, 2000, p. 37). 


Most of the studies are independent studies, totally unrelated to the National Evaluation of the CSRD 
Program. Further, many of the studies are on “comprehensive school reform,” which may have been put into 
place quite independent of the CSRD program. In this sense, the bulk of the studies should not be considered 
evaluations of the CSRD program. For instance, eight of the studies are evaluations of individual research- 
based methods. Many of the studies are being conducted by the U.S. Department of Education’s (ED) 
Regional Educational Laboratories and supported by ED’s Office of Educational Research and Improvement 
(OERI) or Policy and Program Studies Service. 


Unfortunately, neither of these issues has been given sufficient attention in the existing literature (e.g., 
Datnow and Stringfield, 2000). 
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“comprehensive school reform” does not carry the specific connotation that there are nine 
components—although the principles of comprehensive school reform are compatible with 
and may even be comparable in some instances to CSRD’s nine components. 


Second, the reform heritage just traced has been heavily dominated by the existence of 
research-based “models.” A resulting confusion is that the desired reform (and especially 
its comprehensiveness) has been sometimes assumed to be no more than the adoption and 
implementation of an externally-developed, research-based “model.” However, in reality 
the required reform effort must go beyond the “model,” because most of the “models” are 
not comprehensive. They focus on either a specific subject matter (e.g., reading) or a 
particular phase of instruction (e.g., homework), or both; but a school needs to implement a 
reform that embraces the entirety of its operations, covering all grades, all subjects, and all 
students in the school. To this extent, the breadth of successful reform will go well beyond 
the adoption and implementation of most of the available research-based “models.” 


A final clarification of comprehensive school reform also derives from the ongoing 
NAS experience. Originally, NAS considered “break the mold designs” and “design 
teams”—and ultimately “externally developed reform design”—to be the desired agents of 
change. The original concept of “design,” as used by NAS, may indeed have been 
sufficient to capture the breadth of comprehensive reform. “Design” implies an overall 
plan for the comprehensive reform of an entire school, incorporating all aspects of the 
curriculum, school operation, and instruction. However, many of the research-based 
“models” that later emerged were then assumed to be comprehensive, especially because 
many had the characteristic of being externally developed (e.g., CPRE, 1998). In fact, 
many of the “models” were limited to specific grades or academic subjects and alone could 
not support comprehensive reform unless they were integrated into a comprehensive design 
or plan. 


As a result, the present report considers a research-based “model” to be a way of 
dealing with the first of the nine components of the comprehensive school reform 
demonstration (CSRD) program— implementing research-based methods and strategies. 
Further, to avoid perpetuating the use of the word “model” as necessarily implying 
comprehensive reform, this report uses the language of the legislation, or research-based 
“method” or “strategies,” including when referring to such “models” that are externally- 
developed (such as Success for All or Accelerated Schools). Thus, externally-developed 
methods can lead to comprehensive reform, but only if the method requires a school to 
attend thoroughly the other eight CSRD components. However, if the method does not 
lead in a comprehensive direction, a CSRD school must undertake additional initiatives to 
cover the other eight components necessary for the school to meet its CSRD mandate. 
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1.2 THE FIELD-FOCUSED STUDY 


The present report covers the Field-Focused Study. It is one part of the National 
Evaluation of the CSRD Program, which involved multiple related efforts being conducted 
separately by different research organizations (Doherty, 2000).'* The study called for data 
collection from a stratified, random sample of 18 schools from the first two cohorts of 
CSRD awardees. The data collection was based on intensive, field-based efforts: two 
visits per school year, for each of two years. Appendix A contains the detailed design and 
methodology for the study. 


The Field-Focused Study was mainly aimed at describing the links among school 
reform processes, instruction, and other educational activities. The study involved in-depth 
examination of these linkages, thereby limiting the number of schools that could be the 
subject of inquiry. In addition, the schools in the Field-Focused Study also were examined 
as a source of promising reform strategies (i.e., practices, programs, and technical 
assistance or support strategies) to share with other schools (see Chapter 2). The topics 
covered by the study and in this report include: 


1) The progress in implementing the 9 CSRD components by the 
18 schools; 


2) The role of district and state influences in implementing the program; 


3) The apparent pathways or strategies for implementing comprehensive 
school reform; and 


4) The early signs regarding the potential sustainability of the program at 
the school level. 


Because the Field-Focused Study directly investigated the complexity of events 
(including contextual conditions) involved with the nine CSRD components as they might 
occur at an individual school, the study used the case study method as the primary research 


‘For instance, the Southwest Educational Development Laboratory (SEDL) compiled baseline 
information on all CSRD schools. SEDL’s database provided the sampling frame for selecting the sample of 
schools for the Field-Focused Study, to assure that the selected schools represented the entire universe. A 
second project was a large-scale sample survey of CSRD schools conducted in 1998-99, 1999-00, and 2000- 
01 by Westat. The survey was included as part of the broader National Longitudinal Survey of Schools 
(NLSS), also designed by Westat, that followed a longitudinal sample of Title I schools over that same time 
period. Principals and up to six teachers were surveyed in the CSRD sample of schools as well as the NLSS 
sample, and comparable data were collected from both sets of schools. The 1998-99 sample of CSRD schools 
consisted of 318 schools. In 1999-00, the CSRD sample was augmented by 725 schools. The data on the 
CSRD and NLSS schools were analyzed by RAND. 
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method (Yin, 2003).'” To tighten the analytic process, the application of the method to the 
Field-Focused Study included four important features not always found in other case study 
research, as follows. 


A theoretical framework guided the study. The first feature was the use of a 
theoretical framework—i.e., “logic model”—whose development began prior to data 
collection. The evaluation team developed a logic model to depict the working of CSRD at 
any given school (see Exhibit 1-3). The model portrays the 9 components of CSRD as 
boxes in the exhibit, with CSRD Component 4 (“Measurable Goals and Benchmarks”) 
being split into two boxes (4a and 4b). The generic model also contains two important 
concepts not explicitly part of the original 9 CSRD components but that are necessarily 
involved in attempts to improve student performance: school organization (located as part 
of box 4a) and instruction and learning (see the unnumbered box).'° To preserve space, 
the model does not show the contextual conditions—e.g., district and state policies 
affecting the school—although these conditions also have been part of the Field-Focused 
Study. 


The model, as with all logic models, shows all of the boxes, connected by arrows, that 
hypothesizes how comprehensive school reform might produce change in student 
performance (see Box 4b). The dotted arrows are hypothetically related to the research- 
based method defined as CSRD Component 1. The solid arrows are hypothetical but 
nevertheless serve as logical links suggesting how other school actions can lead to effects, 
which in turn become actions for other effects—all part of a chain of events. Key to 
interpreting and using the overall model, therefore, is that data and findings are needed 
about the “arrows,”’’ and not just the “boxes” in the model—that is, “how” and “why” 


‘Investigators assessing a directly comparable initiative—the scale-up phase of the New American 
Schools (NAS)—arrived at the same methodological conclusion, even though NAS funding levels were 
higher than CSRD funding. The investigators argued that the interventions and research questions called for 
“...a replicated case-study approach, with the unit of analysis being the implemented school” (Bodilly, 1998, 
p. 25). [Emphasis added.] 


‘© Any reform undertaking should be proceeded by a needs assessment; and, in fact, some of the 
externally developed methods called for the conduct of a needs assessment prior to adoption. Without a 
needs assessment, it is unlikely that a school would be able to properly design a comprehensive reform, or 
know which method to select (Datnow and Stringfield, 2000). The Field-Focused Study did not address the 
topic of needs assessments because it was not one of the 9 CSRD components. However, the evaluation team 
subscribes to the research that needs assessments are a necessary step in reform efforts, and believes this topic 
should be addressed in future research. 


For illustrative purposes, the arrows have been shown in a linear sequence that does not appreciate the 
full dynamics of events in real life. However, such dynamics were not ignored in the data collection for the 
Field-Focused Study. 
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Exhibit 1-3 


THEORETICAL FRAMEWORK FOR A CSRD SCHOOL 


9. Coordination of 5. Support by 
resources faculty, etc. 


School Organization: 1.* Research - 
4a. Measurable goals and based method 
benchmarks 


4b. Student 
performance 
or strategy 


3. Professional development 
6. Parental and 8. Evaluation 
community involvement 


7. External support and 
assistance 


* = The numbered boxes refer to the nine CSRD components. 


events in one box affected those in another, not just the identity and presence or absence of 
the events within each box. 


In the Field-Focused Study, data collection teams were asked to customize the generic 
logic model into a specific but tentative model for their school(s) prior to actual data 
collection. The details of the logic model were likely to vary, depending upon the nature of 
the research-based method that the school had adopted to satisfy CSRD Component 1, as 
well as other conditions that might have been revealed during a site-screening phase that 
led to the selection of the school for inclusion in the Field-Focused Study in the first place. 
The stipulated logic model then became the occasion for field inquiry—appreciating that 
“discovery” of unexpected events and relationships also could occur as part of the data 
collection procedure. 


The evaluation team could then customize the generic logic model, depending upon 
the CSRD Component 1 research-based method or practice that had been adopted (Success 
for All and America’s Choice in these examples) (see Exhibit 1-4). The labels in the boxes 
reflect the activities emphasized by each method, and the arrows suggest the connections 
implicit in each program. Important in these two logic models are therefore the boxes with 
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Exhibit 1-4 


BEGINNING TO CUSTOMIZE THE ORIGINAL FRAMEWORK: 
TWO DIFFERENT CSRD MODELS 


1-4a: Success for All 


9. Local, state, and federal 5. 80%of faculty vote ; 
funds need to replace by secret ballot to 2. Comprehensive and 


reading program adopt the model aligned design 


4a. Redistribute time and 
schedule so each child 1.* Success 


has 90-minute block for for All: 4b. Student 
reading instruction Reading performance 
in reading 


3. Ongoing 


coaching support 
for teachers 


Prescribed 
instructional 
materials 


8. Existing state and 
6. Family Support Team: local assessments 
Parents engaged in reading evaluate progress 
for 20 minutes each day; 
advisory committee 


7. SFA provides external advisor 
who visits school regularly 


1-4b: America’s Choice 


9. Requires coordination 5. Substantial 


of resources majority of faculty : ! 
not addressed specifically must support model aligned design 


2. Comprehensive and 


4a. May require some 1.* America’s 
adaptation, but not a key Choice: 4b. Improved 
part of the model Comprehensive in student 
areas of instruction performance is 


core 
3. Extensive attention 
to professional Core model is 


development standards-driven 8. Uses new standards 
instruction and reference examinations 
learning and state assessment 
6. Parental and results 
community 
involvement 


7. National center and regional 
coordinators provide training and support 


* = The numbered boxes refer to the nine CSRD components. 
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no connecting arrows—implying that the adopted method does not, on the surface, cover 
all of the CSRD components. To achieve comprehensive school reform, a school must 
therefore find another way of addressing the non-integrated component(s). For example, 
Success for All’s defining feature is that it only covers reading—even though it covers 
reading comprehensively; thus, the program in and of itself does not connect with CSRD 
Component 2 (“Comprehensive and Aligned Design”); America’s Choice’s attention to 
instruction omits any explicit connection to CSRD Component 6 (“Parental and 
Community Involvement”) and Component 9 (“Coordination of Resources”). 


Overall, careful use of logic models strongly enhances the precision and logic of case 
study research, especially in evaluation studies. 


Rival explanations were investigated. A second feature was the explicit identification 
and then collection of data about rival explanations (Yin, 2000). In the case of the Field- 
Focused Study, the rivals were alternative explanations for any of the observed outcomes— 
e.g., whether conditions other than the CSRD funding and mission could account for the 
observed student performance outcomes (positive, negative, or neutral) or even for a 
school’s reform-oriented implementation actions.'® The use of rivals is aimed at reducing 
the uncertainty about potential relationships, even though the reduction will not nearly be 
as large as that occurring in a “true experiment.” Collecting data about possible rival 
explanations was therefore part of the case studies of the CSRD schools. Such data 
collection strengthened the case studies by broadening the data to be collected but keeping 
the entire effort well-targeted. 


Chronological timelines traced key events. A third feature was the explicit use of 
chronological timelines to trace the key events in the schools being studied. The temporal 
sequence of events was examined for clues regarding the strength of CSRD or rival 
explanations. The Field-Focused Study therefore attempted to collect data faithfully on the 
timing of various events. 


Classrooms were observed. A fourth and final feature involved data collection aimed 
at revealing possible differences in implementation patterns across different classrooms 
within each school. If reform actions were incrementally implemented or affected one 
curriculum topic but not another, the targeted classrooms should have shown earlier 
evidence of implementation than the nontargeted classrooms. To put this feature into 
place, the field teams used a formal classroom observation instrument that was part of the 
case study protocol, described next. 


‘8 common misinterpretation in using the rivals method is that the rival is perceived to be that “no 
positive outcome occurred.” Key to avoiding this misinterpretation is that the method calls for defining rival 
explanations (or hypotheses), not rival outcomes. 


1-12 


Site visit teams used a case study protocol. The main instrument for the Field- 
Focused Study was a case study protocol, contained as an attachment to Appendix A, 
which also gives fuller details on the methods used in the entire study. This protocol was 
the subject of ED and OMB review, final clearance for which was received in October 
2000. 


The protocol was designed as an instrument aimed at the investigator, not at any 
group of respondents. The instrument represented a field team’s agenda for collecting 
evidence from a variety of sources—including but not limited to open-ended interviews. 
Using the collected evidence, the protocol’s topics were then addressed by the team by 
creating a formal and documented database that included quantitative as well as qualitative 
data. The focus of the protocol was on tracking and documenting actual behaviors and 
events through converging evidence. This objective may be contrasted with surveys, where 
the information is based on self-reports by a survey’s respondents. 


1.3 SAMPLE OF SCHOOLS BEING STUDIED 


Though small in size, the role of the 18 schools in the Field-Focused Study was 
nevertheless to represent all of the CSRD schools. A two-stage sampling procedure, 
described fully in Appendix A, also attended to stratifications by research-based method 
and by school district. The operational goal was to have two schools with the same 
research-based method (in different districts) and two schools in the same district (but with 
different methods). The stratification provided the basis for a minimal replication design, 
whereby implementation experiences could more readily be attributed either to the method 
or to the district conditions or to neither—thereby helping to arrive at a broader set of 
conclusions than attainable through a set of individualized or idiosyncratic case studies 
alone. 


‘The clearance came just in time to schedule and conduct the first round of site visits in the fall and 
winter of 2000-01, which also included a final training session for the teams held in early November 2001. 
Thus, the teams had to scramble during these few months to cover all the designated schools. 


*°This methodology proved to be extremely effective as the Field-Focused Study progressed. Site visit 
teams were able to crosswalk information and data gathered at schools within the same district, and 
investigate reasons for missing information. For example, staff from one school frequently referenced 
another districtwide research-based method or strategy they credited with allowing them to continue reform 
efforts once CSRD funding ended. When no reference to that initiative was made at the second school in that 
district, the team had enough contextual information to ask probing questions and learn that staff at the 
second school were struggling with internal support issues, as well as potential reorganization, and had not 
bought into the new method. 
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The selected schools were promised anonymity for participating in the evaluation. As 
a result, Exhibit 1-5 only shows the locations of the schools, by state. However, more 
detail about the schools’ characteristics can be found in Exhibit 7 of Appendix A. 


1.4 STUDY ACTIVITIES AND A GENERAL CAVEAT 


The Field-Focused Study was a multi-year effort whose main data collection occurred 
over a two-year period (2000-01 and 2001-02). Findings from the first two rounds of site 
visits, which took place during each of the two semesters in the academic year 2000-01, 
were reported in an annual report to ED in December of 2001 (COSMOS, 2001). The 
current report updates and expands on those findings (Section 2), and draws conclusions 
about the implementation of CSRD (Section 3). As a general caveat in interpreting the 
findings, note that the Field-Focused Study was not designed to determine the causal 
effects of the CSRD program or of the specific research-based methods employed by the 18 
schools. 


Exhibit 1-5 


LOCATION OF THE FINAL SAMPLE OF CSRD SCHOOLS 
FOR INCLUSION IN THE FIELD-FOCUSED STUDY 


2. CSRD IMPLEMENTATION: 
THE 9 CSRD COMPONENTS 


The initial two rounds of site visits during 2000-01 provided preliminary information 
about the status of CSRD implementation at the 18 schools. These findings were updated 
and expanded upon following the final two rounds of site visits in 2001-02. During the 
final year of data collection, five schools had officially completed CSRD and were to be 
operating without CSRD funding” (their CSRD activities began in 1998-99), and 13 were 
in their third and final year of funding (CSRD activities began in 1999-00). The site visits 
collected data about: the progress being made on the 9 CSRD components; salient district 
and state conditions; the likelihood that reform would be sustained; and promising reform 
strategies practices emerging from the sites.~* 


This section discusses observations regarding each of the 9 CSRD components, 
including the use of the research-based method at each school (CSRD Component 1) and 
the breadth and comprehensiveness of the schools’ reform designs (CSRD Component 2). 
Throughout this section are references to promising reform strategies (found in Appendix 
B) investigated during site visits to the 18 CSRD schools. These strategies are provided as 
examples of innovative or successful ways in which schools addressed one or more of the 9 
CSRD components. 


Section 3 of this report then covers related data collection about contextual conditions, 
such as district and state policies possibly affecting comprehensive reform at the schools. 
The section summarizes the overall strength of CSRD implementation, also examining the 
conditions or pathways associated with implementation. The section ends with an analysis 
of the likely sustainability of comprehensive school reform following the end of CSRD 
funding. 


2.1 IMPLEMENTING RESEARCH-BASED METHODS 
(CSRD Component 1) 


All of the schools initially adopted either a nationally or locally developed research- 
based method (see Exhibit 2-1). Among the “national” methods, 3 schools were 
implementing Success for All (SFA) only; 2 each were implementing High Schools that 
Work (HSTW), Accelerated Schools, Lightspan, and Coalition of Essential Schools (CES); 
1 each was implementing Success in the Making (SIM) and Co-NECT and the dual method 


*!Three of the five schools had received no-cost extensions allowing them to carry on some CSRD 
activities for another semester or year. 


*? Appendix D contains short summaries of these reports. 
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RESEARCH-BASED METHODS ADOPTED BY 18 CSRD SCHOOLS AND 


Exhibit 2-1 


FIDELITY ASSESSMENTS OBTAINED FROM DEVELOPERS 


CSRD 
Award 
Status in Fidelity 

School Name/Method* 2001-02 Developer Fidelity Assessments Obtained Rating’ 
School A Accelerated Schools Assessment made by local High 
Accelerated Schools University of Connecticut representative of developer 
School B Southern Regional Ed. Bd.; On-site reports obtained from the Medium 
High Schools That Work Atlanta, GA developer 
School C BBN Corporation Assessment made by local Medium 
Co-NECT Cambridge, MA representative of developer 
School D SFA semi-annual implementation High 
Success for All/Roots & Wings review reports 
School E Developer had no contact with Low 
Coalition of Essential Schools school; school not implementing 
School F Assessment made by local Medium 
Coalition of Essential Schools representative of developer 
School G Assessment made by local High 
“Special Literacy Model”* representative of developer 
School H Assessment made by local Low 
Success for All representative of developer 
School I Lightspan Partnership Self-assessment by school staff (no Low 
Lightspan San Diego, CA contact with developer) 
School J Local superintendent of Assessment made by local Medium 
“Performance Tasks Model”* representative of developer 
School K SFA semi-annual implementation High 
Success for All review reports 
School L Local consultant and school | Assessment made by local Medium 
“Behavior Mod. Model”* leadership representative of developer 
School M Lightspan Partnership Assessment made by local Low 
Lightspan San Diego, CA representative of developer 
School N SFA semi-annual implementation Medium 
Success for All review reports 
School O7 Computer Curriculum Corp.; | Assessment made by school Medium 
Success-in-the-Making Sunnyvale, CA coordinator; developer made no 

fidelity judgements 

School P Southern Regional Ed. Bd.; On-site reports obtained from the High 
High Schools that Work Atlanta, GA developer 
School Q Accelerated Schools Assessment made by local High 
Accelerated Schools University of Connecticut representative of developer 
School R School leadership Developer makes no fidelity Low 


“Comp. Reform Model”* 


judgments 


*Locally developed methods are in quotes and have been given pseudonyms. 


‘Ratings are based on developer enumeration of presence of method components at each school, as well as 


judgment about school’s understanding of the method’s essence. Developers articulated judgments with some degree 


of detail about each school. 


*School O originally identified Co-NECT as its reform method in its CSRD application; it decided instead to 


expand the use of the SIM method. 


of SFA and Roots & Wings; and 4 schools were implementing “locally developed” 
methods.” Appendix C contains descriptions of the methods in the final sample. 


2.1.1 Implementation of the research-based method is fully proceeding at 9 of the 
18 schools, with partial or minimal implementation at the other 9. Using a five-point 
scale (not implementing, planning, piloting, implementing, and fulfilling or sustaining),~* 
15 of the 18 schools were judged to be implementing the method at some level. However, 
the teams’ judgments distinguished between two subgroups: 9 schools (2 of the 9 had 
completed their CSRD grant period) that appeared to be in the implementing or fulfilling or 
sustaining category, and 6 schools (all in their third year of the CSRD grant) still only 
partially implementing (i.e., planning or piloting) their method (see Exhibit 2-2). [Asa 
promising reform strategy, see “Stringent Implementation of the Research-Based Method 
Believed to Result in Improved Reading Score,” in Appendix B.] 


Of the 6 schools judged to be only partially implementing (i.e., in the planning or 
piloting phase) two schools (Schools I and M) adopted Lightspan as their research-based 
method—a method requiring extensive home-parent participation. Both schools had failed 
to achieve such participation, and by the time of the final site visit, the method was in use at 
only one grade level at each school. A third school (School B) was a school whose 
implementation of a method only began to expand in the second year of CSRD funding, but 
by its final year appeared to be diminishing again. The other three schools (Schools C, L, 
and O) were implementing portions of the method schoolwide but had not yet adopted all 
of the components associated with the method. 


Implementation appeared to be minimal or had stalled completely at three schools 
(Schools E, H, and R—all of which had completed their CSRD grant period), as explained 
in Exhibit 2-3. 


2.1.2 All but four schools are implementing the originally proposed method. With 
the exception of 4 of the 18 schools, all had kept to the method named in their original 
CSRD application. Schools E and H were no longer implementing the method they 
proposed for the reasons described above. School R was only using remnants of its locally- 


*°Because the CSRD legislation called for schools to employ research-based methods and strategies that 
have been successfully replicated, most schools in the first and second cohort adopted externally developed 
methods. While locally developed methods could have been entirely developed within the schools, the 
likelihood (confirmed during the Field-Focused Study) was that local methods were developed externally. 


“The scale is derived from Bodilly (1998). To arrive at a score, field teams were instructed to 
triangulate data from three sources: interviews of school staff; direct observations of classroom and school 
behavior; and consultations with the originator or “developer” of the method (who was available for most of 
the schools). The five-point scale does not fully reflect the complexity of method implementation: methods 
can be implemented 1) fully in some classrooms but not at all in others; and 2) partially in some or all 
classrooms. Further, different components of a method can be implemented or not implemented (and in some 
classrooms but not others). 


2-3 


Exhibit 2-2 


IMPLEMENTATION STATUS OF RESEARCH-BASED METHODS 
(Component 1) 


Level of 
Implementation School/Method* Implementation Status 


School A Elements of the method were observed in al/ classrooms; teachers 
Accelerated Schools were actively engaged in accountability and decisionmaking at the 
school, using expertise to affect change. 


School D Method was in use in all classrooms in reading and math; teachers and 
uccess for All students were familiar and comfortable with the curriculum. 
Nearly all faculty were involved in collaboration groups, school 


Coalition of Essential Schools _|established three school-to-work academies (with additional academies 
planned for future years). 


Appears to be 4 [school G Materials and concepts were in use in every classroom and in all facets 
implementing Special Literacy Model”* lof the schools curriculum. 
or “fulfilling and - - 
sustaining” the School J Aspects of the method were in use in al/ classrooms; parts of the 
method (n=9) “Performance Tasks Model’* _|method (i.e., portfolios) were more evident. 
School K Method was in use in all classrooms, for al/ students, every day, though 
uccess for All use in the upper grades was faltering. 
School N A/l classrooms were using the method during the designated time, with 
uccess for All varying levels of expertise. 


School P Use of method strategies was evident in nearly all classrooms; class 
igh Schools That Work periods scheduled in HSTW-recommended blocks. 

School Q Most teachers and administrators supported use of the method, and 

Accelerated Schools laspects were observed in all classrooms. 

School B School began implementation efforts, but support faltered during 2000- 
igh Schools That Work 2001 when staff voted against structural changes. 


Teachers were organized into small communities; students worked in 
interdisciplinary project groups; technology use was improved. 
However, the method was not used equally by all staff. 


No consistent use of the method observed during 2000-01; changes in 


Appears to be implementation plan resulted for 2001-02 and the method was used in 
partially — all 1st grade classrooms (and 80 percent of 1st grade homes). 
implementing 


School L Implementation of behavioral aspects of the method varied by teacher; 


the method (i.e., ; hos of : : 2 : nae 
“Behavior Modification Model™ |implementation of instructional aspects was minimal. 


“planning” or 
piloting”) (n=6) Support for the method was limited throughout CSRD period; despite 
improvements in use in 2000-01, only one class level and an 
afterschool program used the method in 2001-02, and home 


implementation stopped. 


Teachers used method inconsistently, with preparation for state 
assessment taking precedence; implementation became /ess consistent 
in 2001-2002 when state-intervention ended. 


By 2001-2002, implementation had stopped, as the school focused on 
Is not implementing,{Coalition of Essential Schools |organizational and behavioral problems; some aspects of the method 


or ceased remained as part of school philosophy. 

implementing School fully implemented the method throughout CSRD award period; 
the method Voted to shift to district reading curriculum and implementation stopped 
(n=3) in 2001-2002. 


Aspects of the method were initiated, but diminished (almost entirely) 
“Comprehensive Reform after first year of funding; in 2001-2002, some elements were being 
lodel”* revived, but with no attention to any CSRD design. 


*Locally developed methods are in quotes and have been given pseudonyms. 
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Exhibit 2-3 


THREE CSRD SCHOOLS WITH MINIMAL OR STALLED IMPLEMENTATION 
OF THE ORIGINALLY-ADOPTED RESEARCH-BASED METHOD 


School/Method* 


School E 
Coalition of Essential Schools 


School H 
Success for All 


School R 
“Comprehensive Reform Model”* 


Reason for “Not Implementing” Score 


School E underwent significant organizational changes (splitting from a 
middle school and becoming a separate charter high school), and the 
attention and energy of staff shifted between year 2 and year 3 from 
reform to these structural changes. In addition, following the move to a 
new location, the school staff were faced with further school management 
problems— including disciplinary issues with students, as well as the 
general administration of the school. 


During the first two rounds of site visits in 2000-01, the site visit team 
identified School H as fully implementing the method. The loss of CSRD 
funding was accompanied by a shift in district support for the method, a 
greater emphasis in the district and state on student achievement scores, 
and a faculty vote at School H to stop implementing the method during 
the 2001-02 school year. While the school staff continue to use some of 
the method’s practices, the school’s implementation status nevertheless 
shifted to the “not implementing” category. 


School R had started implementing a locally developed, schoolwide 
reform method in its first CSRD year (1998-1999), following several years 
of planning by the school’s top officials and faculty. However, the 
principal who had led the reform planning left School R just before the 
start of the first CSRD year, and the school then had three different 
principals during the three-year period coinciding with its CSRD award. 
During this period, although implementation of the method had begun in 
one grade in 1998-1999, commitment to the method diminished rapidly 
with the succession of principals, and as of 2000-2001 none of the 
method's features were in place. The school obtained a one-year, no- 
cost extension for their CSRD award’ and in 2001-02, and began to 
implement remnants of the original reform method, but without any 
understanding of the relation of those elements to a CSRD reform 
design. The school therefore was still judged to be “not implementing.” 


* Locally developed methods are in quotes and have been given pseudonyms. 
'The school spent only a fraction of its CSRD funds by the end of year 3. The newest principal 
(serving his first year during 2000-01) was committed to revisiting the CSRD activities and the state 


subsequently approved the request. 
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developed method that aligned with district policy and the changing needs of the school. 
School O changed its method from the one originally proposed in its CSRD application 
because, following the submission of the CSRD application, the school received a poor 
performance rating in the state’s accountability system. Because of the poor rating, the 
district encouraged School O to expand its use of an alternative method, which was being 
implemented at many other district schools. With state approval, School O therefore 
switched from its originally proposed Co-NECT method to Success-in-the-Making (SIM).”° 


A fifth school (School N) was implementing the originally proposed method, but the 
field team failed to identify it properly during the screening procedure. School N had been 
expected to adopt Roots & Wings, a method developed by the Success for All Foundation 
to cover mathematics, science, and social studies, in addition to reading. However, 
although School N was later found to have only implemented Success for All (and hence 
was only covering reading), the school was using both the “roots” level and the “wings” 
level of SFA (but in reading only). During the screening, the school therefore responded 
that it was using Roots & Wings when in fact no method was being used outside of 
reading. 


2.1.3 Developers gave mixed fidelity ratings to the schools. The field teams were 
able to obtain information about fidelity from developers for 14 of the 18 schools 
(including locally developed methods).”’ In 6 of these 14 cases, the developers judged the 
fidelity of implementation to be high; in 1 of the 2 judged to be low, implementation also 
was low as judged by the site visit team (the other school was rated as having partial 
implementation). The teams were unable to obtain fidelity ratings from the method 
developers for four schools: two schools were implementing a national method, but had no 
contact with the developer; one school was implementing a locally-developed method and 
had no external consultant; and one school’s developer was not willing to make a 
judgement about fidelity. In all but the last case, the school’s own assessment of their 
fidelity was “low” (the school with the developer unwilling to make an assessment rated 
their own fidelity as “medium”). Exhibit 2-1 also contains the ratings made about the 
fidelity in implementing each method. 


*°The evaluation team became aware of the switch too late to change the final sample of schools to be 
studied. As a result, the final sample, originally to include a pair of Co-NECT schools, had one Co-NECT 
and one SIM. 


*°Contrary to School N’s usage, schools adopting only the reading component generally report that 
they have adopted Success for All. As a result, the final sample—intending to have one pair of Success for 
All (only) and one pair of dual SFA-Roots & Wings combined methods—then had three Success for All (only) 
schools and one school using a combination of the Success for All and Roots & Wings methods. 


°’Most often, the information came as a result of a face-to-face meeting with the developer’s field 
representative who was providing external technical assistance to the school. However, in other cases the 
team either contacted the developer by telephone or reviewed a formal report submitted by the developer to 
the school. 
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2.1.4 Experiences between locally developed and national methods have not 
markedly differed. Of particular interest regarding these methods was the collection of 
locally developed methods, for two reasons. First, at the national level, less is known about 
them than about the “national” methods. Second, one possibility entertained at the outset 
of the Field-Focused Study was that a CSRD school might have shared in the development 
of a locally developed method (e.g., serving as a co-developer), thereby qualitatively 
altering its expected CSRD implementation experience. 


Of the 4 locally developed methods, 3 were similar to national methods in having: a 
formal developer; prior evidence of effectiveness; and ongoing use at two or more schools. 
Proportionately, the implementation experiences by the schools with locally developed 
methods did not differ markedly, as shown previously in Exhibit 2-2, from those of the 
national methods. Further, with the exception noted below, the schools did not have any 
role in developing the methods. Thus the main difference between the national and locally 
developed methods appears to be that the latter were only known and being used within a 
limited regional area—i.e., either a single school district or multiple regions within a single 
state. 


As one result, the field teams were able to locate the local method developers and to 
collect data from them similar to the data from the national method developers. For 
instance, the local developers were providing schools with external technical assistance, 
similar to the assistance being provided by their counterparts with national methods. The 
single exception again had to do with School R, whose locally developed method was a 
truly “home-grown” combination of several practices, none of which itself had any method 
that had a developer.”® 


Two of the methods (adopted by Schools G and J) had both instructional and 
curriculum content (see Exhibit 2-4). For instance, School J’s method involved the 
development and maintenance of a bank of substantive student tasks (“authentic” tasks) 
covering each academic subject and enabling students to develop portfolios of their work. 
The tasks also promoted reasoning and thinking as part of the learning process. In 
addition, local teachers were encouraged to use their classroom experiences to devise and 
nominate new tasks to be reviewed for inclusion in the bank. [As a promising reform 
strategy, see “Research-Based Method Guiding Scope and Sequence of Lessons in All 


*8School R’s CSRD application had indicated that, of the component practices comprising its whole 
“model,” the adherence to state standards represented the research-based component of the entire “model.” 
The state standards go beyond a mere listing of standards and specify a profile of learning—defining what 
students should know, understand, and be able to demonstrate to a high level of achievement. Students must 
then satisfactorily complete a state-prescribed classroom assessment for each standard, to show that they have 
met the standard. However, schools and teachers are to define how students are taught and the specific 
assignments they are required to complete. In this sense, a case can be made that the state’s standards do 
provide guidance similar to that of a research-based method. 
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School 


School G 


School J 


School L 


School R 


Method 
Pseudonym 


“Special Literacy 
Model” 


“Performance 
Tasks Model” 


“Behavior 
Modification 
Model” 


“Comprehensive 
Reform Model” 


Exhibit 2-4 


FOUR LOCALLY DEVELOPED METHODS 


Description 


The method has three goals: 1) to implement effective practices for K-3 literacy learning and 
teaching; 2) to develop teacher leadership teams within a school and its district; and 3) to 
develop mentor teachers to support the preservice education at the university. The method 
is centered around a “special topics” university graduate course. Teachers receive credit for 
their method-related participation. An initial two-week summer institute, attended by 
representatives from the school, helps to plan the activities over the ensuing academic year, 
such as the establishment of teacher teams, parent literacy nights, and parent mathematics 
nights. The method claims to be highly effective in building good instructional practice, 
teacher cooperation, and the use of a specific array of instructional materials. The method is 
based on academic research on early literacy. 


The method is composed of discrete performance tasks, accessible via a Web site, that 
teachers may select and use to deliver instruction. Tasks are available for all content areas 
(though more are available for reading/language arts than for any other area). Tasks are 
designed to be examples of real-life experiences (“authentic” tasks) enabling students to 
acquire content knowledge, good work habits, and thinking and communication skills. The 
method promotes reasoning and thinking as part of the learning process. Examples include 
organizing a healthy luncheon, guiding classmates on a trip, corresponding with a scientific 
author, and composing a song. The method also incorporates student self-assessments for 
judging the quality of a product and overall student performance. Students develop a 
portfolio of their work, and these, along with students’ self-assessments, are used to 
demonstrate and evaluate process and achievement. 


The goal of the method is to achieve schoolwide change enabling “greater academic 
achievement, productivity, and prosocial competency.” The method consists of a behavioral 
protocol, climate-setting strategies, and instructional approaches to improving student 
achievement. Teachers use techniques such as the good behavior game (used during 
regular instruction time to reward good behavior and cite inappropriate behavior) to teach 
students what behaviors to exhibit and what behaviors to inhibit. Other behavior 
modification techniques also are prescribed by the method, including organizing students in 
the classroom into teams, giving students “jobs” within the classroom, and student 
monitoring. In addition, students are taught to walk in “model lines” (or one behind the other, 
with their hands behind their backs) and to use “model voices” (volume levels appropriate to 
the situation). Other elements of the method have included having students graph their own 
performance in spelling and math, so that they learn how to do self-assessments and can 
see their progress, and engage in peer tutoring. 


The original method included the implementation of state standards, which the school 
identified as among the first at the state level to embrace educational quality reform and 
significantly different from other state and national models in that they were more project- 
land performance-oriented.” Other components of the comprehensive method included a 
Freshman Foundation course, intended to help freshman to get prepared for high school; a 
flexible block schedule; an interdisciplinary curriculum, whereby teachers of different 
subjects would have common planning time, hold daily team meetings, and coordinate their 
instructional schedule; the integration of school-to-work (school-to-career) topics; the 
integrated use of technology into the curriculum, especially taking advantage of individually- 
assigned laptop computers; and a vision of the school as having a “global” magnet, 
transcending the four separate magnet programs at the school. In the final year of 
implementation, the method had been revised: 1) implementation of state standards 
continued, 2) the Freshman Foundation course was replaced with Freshman Teams and the 
use of “My Action Plans (MAPs)” by individual students to guide progress through high 
school and toward future goals; and 3) the laptop project ended; and 4) the concept of a 
“global” magnet was replaced with district required small learning communities. 
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Subject Areas,” in Appendix B.] The methods at the two other schools (Schools L and R) 
more closely resembled “process” methods, focusing either on behavior and climate-setting 
strategies (School L) or on the use of technology in the curriculum and Freshman transition 
initiatives (School R). [As a promising reform strategy, see “Teacher Teams Provide 
Students with Cross-Curricular Authentic Learning Opportunities,” in Appendix B.] 


2.1.5 Classroom observations are consistent about ongoing experiences with the 
methods. Direct observations of classrooms in all of the 18 schools were an integral part 
of assessing the extent and nature of implementation.” In only a few cases was the 
observed implementation different from that expected as a result of the interviews of the 
schools’ officials. For instance, Schools J and B had individual classrooms where 
implementation was much lower than expected based on the officials’ interviews. In the 
opposite direction, School F’s classrooms showed a high degree of implementation even 
though the interviews had described implementation as still “in progress.” 


More difficult to assess through classroom observation was any potential role of the 
other CSRD components. Brief interviews with the teachers whose classrooms had been 
observed proved more informative. The teachers reported about: their professional 
development experiences, their perceptions of staff support for reform; and apparent parent 
and community support in conjunction with their classroom activities. 


2.2 COMPREHENSIVENESS OF DESIGNS FOR SCHOOL REFORM 
(CSRD Component 2) 


The field teams assessed the extent and quality of comprehensiveness by attending to 
the presence of the eight remaining CSRD components and their apparent linkage. Key 
among these other 8 components is Component 2 (evidence of a “comprehensive design”), 
which the teams assumed needed to exist if implementation was later to achieve actual 
comprehensiveness. 


To assess progress on this component, the field teams first tried to obtain any 
documentation of such designs—e.g., the breadth and scope of any school improvement 
plan, or even the apparent comprehensiveness and convergence of a school budget that 
integrated CSRD funding within it.°? The teams then also tried to determine the school 


*°Fach classroom observation involved the use of a formal classroom observation instrument (included 
in the case study protocol—an attachment to Appendix A). 


*°Not considered sufficient evidence of a comprehensive design was sheer existence of the school’s 
original CSRD application alone. All of the applications address the nine components because states required 
their districts and schools to do so. However, addressing these components did not necessarily mean that 
they had been adequately operationalized (and, in some cases the application had been totally prepared by the 
district and therefore may not have become a salient part of the school’s understanding of CSRD). To be 
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staff’s level of awareness and understanding of reform, independent of whether such 
documentation existed or not. On both fronts, the teams’ inquiry was aimed at determining 
whether the school understood that CSRD meant reforming the entire school—i.e., 
potentially expanding to its entire way of doing business and all its operations—and not 
simply adding a new function or project or activity; and that in carrying out such reform, a 
school would necessarily have to attend to the alignment or connectivity among all of its 
operations. 


2.2.1 Less than half of the 18 schools had reforms with a comprehensive design. 
Eight of the 18 schools (Schools A, B, D, F, G, J, L, and P) satisfied both conditions: 
a) school reform plans or documentation that were comprehensive, and 
b) acknowledgments or understandings of comprehensiveness by the school staff who had 
been interviewed (see Exhibit 2-5). Six other schools (Schools C, H, K, N, O, and Q) had 
comprehensive plans or documentation, such as school improvement plans, but staff were 
unaware of or detached from any comprehensive vision associated with such 
documentation. Where understandings of comprehensiveness fell short, the general reason 
was that CSRD breadth or implementation strongly resembled a separate project or even 
“add-on” activity at the school. Among most of the remaining schools, neither 
comprehensive documentation nor requisite staff understanding was present. 


2.2.2 Comprehensiveness of CSRD can emerge from either working with 
comprehensive research-based methods or having a comprehensive school improvement 
plan. Of the 8 schools that had satisfied both conditions, the comprehensive designs for 5 
of them (Schools A, B, F, J, and P) were associated with the comprehensiveness of the 
research-based methods that had been adopted, but not necessarily because the school had 
developed any separate comprehensive reform vision or plan; for the other 3 schools 
(Schools D, G, and L), the comprehensiveness was associated with a required school 
improvement plan that was comprehensive even if the research-based method was not. [As 
a promising reform strategy, see “School Improvement Planning Process Aligns Reform 
Initiatives,” in Appendix B.] 


Overall, 10 of the 18 schools may not yet have fully embraced the principles of 
reform and steps needed to achieve comprehensive reform. Compared to the proportion of 
schools implementing research-based methods (CSRD Component 1), the proportion of 
schools with comprehensive designs (CSRD Component 2) was substantially lower. 


credited with pursuing a comprehensive design, schools had to show that they were actually using their 
CSRD applications in planning and implementing reform, and further that the staff interviewed could express 
some awareness and understanding of the comprehensive nature of the activities. 
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Exhibit 2-5 


EVIDENCE OF COMPREHENSIVE DESIGN 


School/Method* 


School A 
Accelerated Schools 


School B 
High Schools That Work 


School C 
Co-NECT 


School D 
Success for All/Roots & Wings 


School E 
Coalition of Essential Schools 


School F 
Coalition of Essential Schools 


School G 
“Special Literacy Model”* 


School H 
Success for All 


School I 
Lightspan 


School J 
“Performance Tasks Model”* 


(Component 2) 


Evidence of Comprehensive Design 


Interviews, classroom observations and documents all made clear that the staff 
understood and strongly supported the reform process, including some 
familiarity with the nine CSRD Components. 


The reform strategies were integrated into the school improvement plan. The 
method and plan appeared to be well understood by staff, students, and 
parents, though support for some elements was declining. 


The school had a single plan, required by the district, linking all activities to goals, 
objectives, benchmarks, and strategies. The plan served as funding 
application for CSRD and was aligned with state standards. However, staff 
were reportedly disappointed with the plan, which they believed failed to 
capture all activities and funding streams. Co-NECT was recognized staff as 
the only comprehensive reform strategy. 


The district required the school to develop a comprehensive plan aligned to the 
district’s vision for improving student performance. Much of the district plan 
paralleled the CSRD reform components. Many of the strategies in the plan 
were specifically tied to the reform methods. Evidence that the staff supported 
the plan were visible throughout the school, particularly in display materials 
posted on walls in classrooms and gathering areas. 


By the final year of funding, the staff had no vision for comprehensive school 
reform, probably due to its pre-occupation with its own survival and 
restructuring. While many staff shared a common philosophy about reform, the 
methods or means for accomplishing this vision were a source of conflict. 


The CSRD reform plan encompassed all 9 CSRD components, though the school 
improvement plan focused solely on goals to increase student performance. 
The majority of staff clearly understood and supported the reform plan. 


The district-required school improvement planning process played a central role 
in establishing the comprehensiveness of the reform effort. Every faculty 
member signs the plan to indicate an understanding of his or her role in 
implementing the plan. 


The school improvement plan had a comprehensive set of goals and strategies, 
generally aligned with the CSRD components, by driven by the state 
assessment. The comprehensiveness of the reform effort was widely 
understood, though by 2001-02, the staff felt their reform efforts were in conflict 
with the districtwide reform agenda and plan. 


The school had two plans, one covering achievement goals, and another 
summarizing a variety of school initiatives. Neither offered a comprehensive 
picture or plan for the nine CSRD components. Teachers described the school 
as having a split focus and a mission that was way to broad. 


The school’s improvement plan did not address the CSRD reform components, 
but did reflect the school’s goals and strategies for improving student 
achievement. The method itself included all reform components, and staff 
appeared to understand and support both the reform effort and the method. 


*Locally developed methods are in quotes and have been given “pseudonyms.” 


(Continued on next page) 
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Exhibit 2-5 (Continued) 
School/Method* 


School K 
Success for All 


School L 
“Behavior Modification Model”* 


School M 
Lightspan 


School N 
Success for All 


School O 
Success-in-the-Making 


School P 
High Schools That Work 


School Q 
Accelerated Schools 


School R 


“Comprehensive Reform Model”* 


Evidence of Comprehensive Design 


The school improvement plan addressed each component in relation to reading, 
but little emphasis was placed on other subjects. A steering committee 
managed implementation and integration of reform components. Staff 
appeared to understand reform effort. 


The district required the school to develop a comprehensive plan aligned to the 
district’s vision for improving student performance, which paralleled the CSRD 
reform components. Staff appeared to understand the district reform plan, 
which was comprehensive (even though the research-based method did not 
appear to be well-integrated into this plan). 


The school’s original improvement plan included Lightspan, but didn’t cover the 
other reform components. A revised plan was finalized in June 2001 and was 
intended to be aligned with state and district standards, however, a new school 
administration the following year did not think the plan addressed the school’s 
needs sufficiently. Some staff believed the plan was being ignored and others 
argued it was not a practical document. 


All 9 elements of the reform were present in the school. A new plan was adopted 
in 2001-2002 that aligned the plan to the state assessment, and set strategies 
for reaching improvement goals. However, it did not appear to be a working 
document at the school; programmatic and other decisions were made without 
reference to the overall design. 


The school’s improvement plan listed its objective as improving student 
performance on state assessments; the plan contained references to a number 
of reform components. The breath of the planning process and SIM appeared 
to provide a unifying plan of action throughout the school. 


The reform strategies were integrated into the school improvement plan. Staff 
and the core leadership team understood and supported the method and the 
reform plan. 


The school addressed the nine reform components in their proposal, with varying 
levels of detail. Some staff, particularly on the leadership committee supported 
the plan, but high turnover among staff resulted little understanding or support 
for the comprehensive reform design by a large number of newer staff. 


The method initially reflected comprehensive whole school reform; but was not 
used after the first year of CSRD funding. During the school’s final CSRD 
year, the reform effort was revived, but the lack of a clear and comprehensive 
plan (understood by all staff) was limiting the impact of the changes. 


*Locally developed methods are in quotes and have been given “pseudonyms.” 
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2.3 PROGRESS IN IMPLEMENTING THE 7 OTHER CSRD COMPONENTS 


As with the first two components, the site visit protocol called for covering each of 
the other seven CSRD components individually. However, the findings from the site visits 
suggested that these seven components could be dealt with in several combinations, in the 
interest of streamlining the discussion: 


e Professional development (Component 3) and external technical 
support and assistance (Component 7); 


e Measurable goals and benchmarks (Component 4) and evaluation 
strategies (Component 8); 


e Support within the school (Component 5) and parent and 
community involvement (Component 6); and 


e Coordination of resources (Component 9) and sustainability. 


To be noted is that the last combination joins one of the seven components (Component 9) 
with another topic of high interest but that has not been regarded as a separate CSRD 
component—sustainability. Inherent to comprehensive school reform is that efforts cannot 
be short-lived as in a specially funded “project.” Thus the Field-Focused Study has 
devoted explicit attention to the issue of sustainability (Section 3.5 addresses the issue of 
sustainability in greater detail). 


2.3.1 Both professional development (Component 3) and external technical support 
and assistance (Component 7) have largely been devoted to the implementation of the 
research-based method, not necessarily comprehensive reform more broadly. The 
rationale for pairing these two CSRD components is as follows: Most schools were 
engaged in moderate to intensive professional development in relation to implementing 
CSRD, but most if not all of this professional development was being provided by an 
external technical support team (see Exhibit 2-6).*' 


Exhibit 2-6 summarizes the professional development reported in the site visits. The 
exhibit distinguishes between method-related professional development and other, reform- 
related professional development. The last column in the exhibit also comments on any 
unusual barriers to professional development identified through the site visits. 


*!By design and instruction from the U.S. Department of Education, and consistent with CSRD 
program guidance, the Field-Focused Study deliberately defined “external” support and assistance to exclude 
assistance by the district. 
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School Name/ 
Method* 


School A 
Accelerated Schools 


School B 
High Schools That 
Work 


School C 
Co-NECT 


School D 
Success for All/Roots 
& Wings 


School E 


Coalition of Essential 


Schools 


School F 


Coalition of Essential 


Schools 


School G 
“Special Literacy 
Model”* 


Exhibit 2-6 


PROFESSIONAL DEVELOPMENT ACTIVITIES 
(Component 3) 


Types and Frequency of 
Method-Related Professional 
Development (PD) Activities 


Extensive training in AS philosophy and 
“powerful learning” strategies for all 
staff; weekly cadres meetings allowed 
staff to identify issues of concern for 
entire school and develop hypotheses 
to address them. 


CSRD increased PD activities by 12 
times compared to year prior to grant; 
all activities were focused on HSTW 
and included peer-to-peer 
presentations. 


Professional development focused on 


comprehensive planning in relation to 
the SIP process, as well as use of 
student achievement data for 
planning, technology integration, 
project-based learning, and 
assessment. 


Virtually all professional development 
was linked to the reform method; the 
principal estimated that 40 hours of 
professional development activities 
were offered per teacher, and that 
about 70 percent of staff participated 
at this level. 


Formal professional development related 
to the method was primarily available 
to the high school teachers during the 
first year of the CSRD grant; little 
occurred in later years. 

Collaborative groups and action teams 
were the main PD-delivery method, 
and nearly all staff participated by 
2001-02; staff took part in a week-long 


“CES change process” course and ina 
local HSTW network. 


Nearly all staff participate in method- 
related PD that is related to 
comprehensive reform and is results- 
based. 


Other Professional Development 


Activities Barriers 


Staff received training in 
successful reading strategies 
for LEP students from local 
university consultants, as well 
as test preparation and 
leadership skills. 


Implementation of Making 
Schools Work at the middle 
schools involved some training 
for high school teachers, 
related to facilitating the 
transition from middle to high 
school. 


Teachers had a common 
planning period to facilitate 
cooperative learning. 


District-wide budget cuts 
would limit formal PD 
activities in 2002-03. 


Nearly one-third of the 
teachers quit or retired; 
support for HSTW was 
fading, though upcoming 
merger with county high 
school may revive the 
reform effort. 


Professional development 
requirements were not 
tied to teacher re- 
certification, giving 
teachers less incentive 
to participate. 


All other schools in the 
district used district- 
promoted reading and 
math curricula, and 
received district- 
sponsored professional 
development; isolating 
school staff. 


Beginning in 2001-02,the school |PD sessions and staff 
closed early two days a month] planning time were being 
to allow staff to address used to address 
schoolwide concerns. administrative issues, 
such as discipline. 


Lack of funding for 
substitute teachers in 
2001-02 may mean 
groups can no longer 
meet during the school 
day. 


Administrators and staff meet bi- 
monthly to work on curricular 
strategies; receive training 
from district on use of data. 


Training by district consultants 
has included: using electronic 
portfolios, using independent 
thinking, distance learning, 
and state standards/ testing. 


Twice monthly PD related to 
math is presented by a math 
specialist. 


*Locally developed methods are in quotes and have been given pseudonyms. 


(Continued on next page) 
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Exhibit 2-6 (Continued) 
Types and Frequency of 


School Name/ Method-Related Professional 
Method* Development (PD) Activities 
School H Until 2001-02, school staff received 


Success for All method-related PD twice per year 
from developer, and on an ongoing 
basis from campus instructional 
coordinator (through modeling, 
observations, and grade-level 


planning). 


School I Earlier training in the method was limited, 

Lightspan but in 2001-02, training by in-house 
staff and accessibility to materials 
better enabled 1st grade teachers to 
begin implementation. 

School J All but three teachers received PD 


“Performance Tasks 
Model”* 


related to the method, including 
training sessions on active learning, 
self and peer assessment, 


cooperative planning, and alignment 
with state standards. 


School K 
Success for All 


Extensive PD offered around method, 
including five-day introductory 
workshop, as well as opportunities for 
training of tutors and the Family 
Support Team. 


School L Staff received training in the method 

“Behavior through in-service days at the start of 

Modification the year, annual retreats, and ongoing 

Model”* modeling and observations by the 
lead teacher. 

School M Activities were primarily focused on 

Lightspan grade 3—5 teachers who used 


Lightspan; one presentation was 
made to all teachers in the school. 
Four Lightspan training events were 
provided during 2000-01; no training 
was provided in 2001-02. 


Other Professional Development 


Activities Barriers 


The district provided training-of- 
trainer PD on content 
standards, access to 
resources, and student 
expectations. When the 
school stopped implementing 
SFA in 2001-02, the district 
also provided training on the 
reading curriculum. 


School provided staff with 
collaborative time during 2001- 
02, allowing staff to meet by 
grade level, examine student 
achievement scores, develop 
ideas and share plans. 


Teachers had to complete 2 half- 
day training sessions to meet 
state’s continuing education 
requirement; district provided 
PD on understanding and 
using student achievement 
data. 


District required teachers to 
complete 2 half-day training 
sessions to meet state’s 
continuing education 
requirement; district provided 
PD related to understanding 
and using student 
achievement data. 


Teachers met bi-weekly (once onjLack of implementation of 
staff issues; once on curricular | instructional portions of 
topics); district required 18 the method indicated 
hours of PD activities, and inadequate PD on this 
provided PD on the math and component. 
reading curricula. 


The district provided between 
three and four PD days a year 
through a catalog of staff 
development opportunities; 
there was no coordination to 
ensure staff selected PD 
sessions related to the 
school’s reform efforts. 


District restricts early 
school dismissal and 
teacher planning days. 


*Locally developed methods are in quotes and have been given pseudonyms. 
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Exhibit 2-6 (Continued) 


School Name/ 
Method* 


School N 
Success for All 


School O 
Success-in-the- 
Making 


School P 
High Schools that 
Work 


School Q 
Accelerated Schools 


School R 
“Comprehensive 
Reform Model”* 


Types and Frequency of 
Method-Related Professional 
Development (PD) Activities 


Method-related PD received twice per 
year from developer, and on an 
ongoing basis from campus 
instructional coordinator (through 
modeling, observations, and grade- 
level planning) 


Teachers participated in monthly 
activities related to the method; CSRD 
funding allowed inclusion of 
kindergarten and Grade 5 teachers; 
training included how to incorporate 
the method into the classroom and 
how to use data. 

All teachers had a minimum of 6 PD 
days from HSTW framework; 75% of 
teachers opted to participate in more 
activities. 


Each staff member received an average 
of 6 PD days; 90% was reform-related. 
School used a training-of-trainers 
approach; topics covered the specific 
instructional and pedagogical 
practices consistent with AS. 


CSRD funds were used in the Year-1 for 
method-related PD, including funds for 
technology-focused training; funds 
were devoted to PD activities in 2001- 
02 for both the development of small 
learning communities and freshman 
teams. 


Other Professional Development 
Activities Barriers 
District provided PD 
related to the district- 
supported reading 
curriculum, which 
School N was not using. 


The district provided training-of- 
trainer PD on content 
standards, access to 
resources, and student 
expectations. 


District provided PD on 
curriculum goals, co-teaching 
methods, using the SFA 
curricula, instructional 
strategies in reading and 
writing, and magnet school 
curriculum. 


Staff participated in PD related to 
vocational education, as well 
as PD related to the transition 
students from middle to high 
school through Making 
Schools Work (Supported by a 
2000 CSRD grant to the 
district middle schools. 


The school received training on 
teaching children of poverty. 
Incentives for faculty 
participation in PD included 
tuition/expense reimbursement 
and continuing education units. 


The school’s focus was on 
aligning teachers’ individual 
PD plans with school and 
district plans; the district had a 
catalog of course offerings 
available to school staff. 


District placed restrictions 
on early dismissals and 
teacher work/planning 
days. 


Budget cutbacks have 
greatly reduced the 
amount of professional 
Opportunities that the 
district can make 
available to schools. 


*Locally developed methods are in quotes and have been given pseudonyms. 
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The first column of Exhibit 2-6 shows that, except in the case of three schools 
(Schools E, M, and R), the schools all have been participating in extensive professional 
development, mainly led by the external method’s developer. To their credit, the 
developers have been able to provide a broad and intense variety of assistance, including: 
1) participation in national conferences and workshops; 2) convening of local workshops 
for the school’s faculty; 3) organizing of annual retreats for the faculty; 4) convening of 
all-staff meetings; 5) specific training to individual teachers; and 6) periodic and in some 
cases frequent on-site support and problem-solving. [As a promising reform strategy, see 
“Professional Development Activities and Instructional Practice Driven by Student 
Assessment Measures,” in Appendix B.) 


The second column of Exhibit 2-6 shows that, although the field teams were to 
inquire about professional development not related to the research-based method, not a lot 
of such professional development was occurring. In 15 of the 18 cases, the broader 
professional development was: 1) being provided by the district, 2) embedded in the job 
structure, or 3) informal. [As a promising reform strategy, “Staff Utilizes Common 
Planning Period as Professional Development Opportunity” in Appendix B.] In 2 of those 
15 cases, as well as in the remaining 3 schools, staff members participated in professional 
development related to the schools’ adoption of a second research-based method. 


Directly corresponding to these observations about professional development, nearly 
all of the 18 schools: a) had external technical support or assistance, and b) derived such 
assistance from the developer of the research-based method (see Exhibit 2-7). 


The intensity and breadth of the assistance in part depended upon the amount of 
CSRD funds allocated for such assistance by the school. Although precise budget data 
were not always available, the site visit team observed that in most cases the bulk of 
schools’ CSRD funds were used to support external assistance from the developer.” Asa 
result, schools that adopted research-based methods requiring ongoing annual outlays 
(compared to outlays for startup years only) were especially likely to need to identify new 
sources of funds, in order to continue comprehensive school reform activities beyond the 
period of CSRD funding. 


Exhibit 2-7 also shows the diversity of the external assistance and support activities, 
ranging from participation in national meetings to teacher workshops to individualized 
(within classroom) on-site support. In reviewing these data, it is important to note that the 
site visit protocol was not limited to external assistance by the method’s developer. Rather, 
to cover CSRD Component 7, the protocol guided the site visit teams to inquire about any 
type of external assistance. As a result, in many cases the teams did find such assistance 


*°This does not necessarily mean that the funds went directly to the developer. For instance, funds 
might have been used to pay for teacher substitutes so that the teachers could participate in the developer’s 
workshops. 
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Exhibit 2-7 


EXTERNAL TECHNICAL SUPPORT AND ASSISTANCE 
(Component 7) 


Types and Frequency of 
School Name/ Types and Frequency of Method-Related Other Reform-Related 
Method* External Support and Assistance External Support and Assistance 


School A 
Accelerated Schools 


School received extensive on-site support from |Consultants provided support in reading 
the developer since formal adoption of the and writing instruction and classroom 
method; by Spring 2002, the principal was and time management. 

concerned that her staff had training needs 

beyond the capacity of the national center. 


School B Developer conducted two site visits per year, |During the final year of CSRD funding, 
High Schools that Work as well as onsite workshops, sponsored staff received training in Making 
visits by schools staff to high achieving Schools Work, an initiative to better 
schools, hosted an annual conference, and prepare students for the transition to 
conducted monthly telephone conferences high school. 
with mentor site leaders. 


School C Onsite Co-NECT consultant serviced School C |The school was implementing a number 
Co-NECT and five others, was available on-site almost of other initiatives (particularly 
weekly, taught classes (modeling) and made reading programs), for which staff 
presentation to all-school assemblies. received training. 
Teams from other Co-NECT schools 
conducted annual site visits. Developer- 
supported “Exchange” website as resource 
for school staff. 
School D SFA Foundation provided extensive training for |External assistance provided by a local 
Success for All/Roots & both the reading and math curricula in the consultant hired by the district to 
Wings first year of implementation. Refresher support the district’s CSRD schools. 
training sessions took place annually, along She helped the school identify 
with visits to assess the level and fidelity of technical assistance providers and 
implementation (twice per year). develop classroom planning guides, 
and was available during monthly 
phone-calls with the principal. 
School E After the first year of funding, the school had No other external support identified. 
Coalition of Essential limited contact with the method developer 
Schools (following 1998-99, the school was not an 
official CES site and did not pay annual fees 
to any CES organization. 
School F Week-long training by CES designed to Extensive training from Annenberg and 
Coalition of Essential perform a year-long activity for visualizing NSF on using collaborative inquiry 
Schools and implementing plans for change. and action research; HSTW network 
meetings every two months to share 
ideas. 
School G The developer conducted a summer institute |The developer also worked with staff to 
“Special Literacy and graduate course work, and was on-site design and develop professional 
Model”* twice per year for in-service sessions. A development workshops based on a 


regional project liaison for the developer was | local needs assessment process. 
on site twice per month, working with 
teachers and providing in-class modeling. 


*Locally developed methods are in quotes and have been given pseudonyms. 
(Continued on next page) 
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Exhibit 2-7 (Continued) 


School Name/ 
Method* 


School H 
Success for All 


School I 
Lightspan 


School J 
“Performance Tasks 
Model”* 


School K 
SFA 


School L 


“Behavior Modification 


Model”* 


School M 
Lightspan 


Types and Frequency of Method-Related 
External Support and Assistance 


Until 2000-02, the method developer visited 
twice a year and provided training and TA on 
SFA implementation. The school did not 
complete a contract with SFA for 2001-02 
when staff voted to shift to the district- 
endorsed reading curriculum. 


During the first two years of CSRD funding, the 
developer conducted one annual site visit to 
observe classes and provide demonstrations 
of how Lightspan could be used in the 
classroom. In 2001-02, the school 
discontinued its relationship with the 
developer and relied on an internal staff 
person to conduct staff training. 


The school worked with the developer during 
Year 1 who provided initial training. In years- 
2 and -3, the school worked with an external 
consultant for the developer, who provided 
training focused on the development of new 
tasks. 


The developer provided training in the method 
for all staff, as well as for members of family 
support team. Staff complained that the 
developer did not respond to concerns about 
appropriateness SFA for students in 
intermediate grades, which may impact 
sustainability of the method at this school. 


Developer provided consistent and strong 
assistance since the inception of the 
program, including helping the school select 
and customize the reform method, 
conducting introductory and follow-up 
training for the method, visiting classrooms, 
serving as facilitator at staff retreats, and 
generally assisting as requested. The 
developer also hosted bi-annual networking 
dinners for local principals. 


In 1999-00, the developer provided support at 
the beginning of the school year. In 2000— 
01, the school expanded the contract to 
include monthly visits to assist in 
implementation. In 2001-02, the new 
administrators at the school decided not to 
renew its contract with the external 
developer. The school relied on internal staff 
to train new staff members. 


*Locally developed methods are in quotes and have been given pseudonyms. 
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Types and Frequency of 
Other Reform-Related 
External Support and Assistance 


A local university provided student 


teachers who supported reform by 
tutoring students. 


Due to state budget cuts, the district 
sought assistance from its regional 
service center. 


School received assistance from 
reading consultants, and district 
assessment staff, as well as “critical 
friend” mentoring support (Sponsored 
by the state for schools rated as in- 
need-of-improvement). 


No additional training from external 
providers was identified. 


No additional training from external 
providers was identified. 


No additional training from external 
providers was identified. 


(Continued on next page) 


Exhibit 2-7 (Continued) 


Types and Frequency of 
School Name/ Types and Frequency of Method-Related Other Reform-Related 
Method* External Support and Assistance External Support and Assistance 

School N The developer conducted implementation visits |School also received training in use of 
SFA to the schools twice annually. On-site other reading initiatives (Accelerated 

support included training in use of SFA, Reader and state-created program). 

conducting classroom observations, and 

customizing training to meet district- and 

school-specific needs (such as the alignment 

between state standards, the state 

assessment, and the method). The school 

reduced the number of visits in 2001-02, as 

CSRD funding dwindled. 
School O The developer provided support (ranging from |No additional training from external 
SIM 0.5 to 3 days each month) on technology providers was identified. 

planning and grade level meetings, made 

presentations on the method during 

education conferences, and assisted in data 

collection review and analysis. 
School P The developer provided two technical review {During the final year of CSRD funding, 
HSTW visits per year, site specific development staff received training in Making 

activities (around the method), visits to high Schools Work, an initiative to better 

achieving schools, monthly teleconferences prepare students for the transition to 

with site leaders, and assessment and high school. 

analysis of student achievement data. 
School Q The developer provided training to school No additional training from external 
Accelerated Schools leadership and staff on-site, conducted providers was identified. 

technical assistance and on-site monitoring 

every six weeks, and developed an 

evaluation plan. In 2001-02 (the school’s 

final year of CSRD funding), the developer 

spent less time at the school, and 

encouraged staff to develop internal capacity 

to support the method. Staff reported more 

assistance in Year 3 was needed, 

particularly for new teachers. 
School R The method was so customized that no single |No other external support identified. 
“Comprehensive Reform | “developer” or external technical assistance 
Model”* team was relevant. 


*Locally developed methods are in quotes and have been given pseudonyms. 
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being provided by parties other than the developer (Schools A, B, C, D, F, G, H, J, N, and P). 
However, within this subset, only rarely did the assistance address comprehensive school reform 
independent of the method. 


The field teams concluded that perhaps the previously observed lower proportion of 
awareness and design for comprehensive reform—independent of the research-based method that 
a school has adopted—was in part because few consultants were available to provide this type of 
assistance,*° and further, because schools have not understood the need to seek such assistance. 


2.3.2 Both measurable goals and benchmarks (Component 4) and evaluation strategies 
(Component 8) have been devoted to tracking student performance, not necessarily 
implementation progress. Having measurable goals and benchmarks (Component 4) enables a 
school to monitor the progress of a school’s CSRD initiative. The actualization of any 
monitoring or evaluation activity, however, requires evaluation strategies (Component 8) and 
planning. For this reason the two topics are covered together. 


For measurable “goals,” all but one of the 18 schools (School E, which became a new 
charter school in 2001-02) identified detailed student performance goals (see Exhibit 2-8).4 
However, rather than considering “benchmarks” as markers of implementation progress or other 
hypothetical steps (e.g., increased enrollment, hours spent, or time on task) needed to achieve the 
goals, many of the schools simply defined “benchmarks” by providing yearly breakdowns of the 
incremental performance gains that would extrapolate to meeting the overall performance goals. 
[As a promising reform strategy, see “Method-Recommended Curriculum Team Aligning Scope 
and Sequence to Provide Greater Opportunities for Interdisciplinary Projects,” in Appendix B.] 
The second column of Exhibit 2-8 shows a small group of schools (Schools B, C, G, J, O, and Q) 
that had defined and were actually monitoring implementation benchmarks, apart from any 
monitoring being done by the external developer. (As a promising reform strategy, see “External 
‘Critical Friends’ Group Provides Objective Assessment of Progress Towards Goals and 
Benchmarks,” in Appendix B.] For instance, the original language of Component 4 does focus 
on student performance and is readily reinforced by the attention that has been devoted to student 
performance in judging school performance. 


*°For instance, no references during the site visits were made to assistance from the U.S. Department of 
Education’s regional educational laboratories, which might have been in a position to deal with comprehensive 
school reform and not just the implementation of a specific research-based method. 


“Exhibit 2-8 merely indicates the existence of these goals and benchmarks, but does not list them. The field 
teams collected the actual lists, which are contained in the teams’ archival databases. The complete lists were too 
lengthy to include in this report, as most of them address target scores or percentage improvements for each grade 
level, for different academic subjects, and, in some cases, for different annual periods of time. 
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Exhibit 2-8 


GOALS AND BENCHMARKS (Component 4) AND 
EVALUATION STRATEGIES (Component 8) 


Benchmarks 


School Name/ 

Method* Student Achievement 
School A Starting in 2001-02, 
Accelerated school developed 
Schools benchmarks to meet 

state proficiency 
levels by grade and 
subject 
School B Included extensive 
High Schools coverage of student 
That Work performance on 
assessments and 
toward graduation 
requirements 
School C Benchmarking plans 
Co-NECT pointed to 
performance on state 
mandated tests 
School D School expected to 
Success for All reach benchmarks 
established by the 
district targeting 
student performance 
(reading readiness, 
attendance, reading 
and math proficiency) 
School E In 2001-02, the 


benchmarks and 
goals were unclear, 
after the school 
restructured asa 
charter school 


Coalition of 
Essential Schools 


School F 
Coalition of 
Essential Schools 


State establishes 
benchmarks for 
student achievement 
scores incorporated 
annually in SIP 


Implementation 


Accelerated Schools 
Assessment Toolkit allowed 
school to develop 
benchmarks; process had 
not begun 


Leadership team developed 
process benchmarks and 
indicators and reviewed 
progress annually 


Co-NECT benchmarks 
included shared 
accountability, project- 
based learning, 
comprehensive 
assessment, team-based 
school organization, and 
use of technology 


Implementation and fidelity 
reviews conducted by 
external developer 


No implementation 
benchmarks 


School met first set of 
implementation 
benchmarks, related to 
participation in collaborative 
groups and development of 
academies, but did not 
establish new benchmarks 
pertaining to full reform 


*Locally developed methods are in quotes and have been given pseudonyms. 
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Evaluation 


Plan 


No formal plan; school 
used AS school 
evaluation 
questionnaire; other 
plans developed by 
individual cadres 


State conducting a 
formal evaluation of 
CSRD 
implementation 


No formal external 
evaluation; self- 
assessment for SIP 
process; and CO- 
NECT processes to 
measure 
implementation 


District contracted 
with a consultant to 
conduct evaluation, 
but turned 
responsibility over to 
schools in 2001-02 


No recent contact with 
external evaluator 


District intends to 
conduct an impact 
assessment (using 
survey and student 
achievement data); 
no evidence of 
written plan 


Activities 


No written findings from 
cadres, who analyzed 
student achievement data, 
and developed plans to 
address deficiencies 


Formal instrument used to 
evaluate instruction during 
classroom observations 


Conduct of parent, teacher, 
principal, and student 
surveys; review of 
achievement scores; 
conduct of focus groups; 
reports from external 
technical assistance 
providers 


End-of-year report 
contained findings 
mandated by district; SFA 
made bi-annual visits to 
the school to monitor 
implementation; no 
evidence of formal 
evaluation report 


Prior activities of an 
external evaluator were 
lost during the school’s 
transition; no activities 
were planned for 2001-02 


School completed surveys; 
district analysis of data in 
progress; no written finding 
available 


(Continued on next page) 


Exhibit 2-8 (Continued) 


Benchmarks Evaluation 
School Name/ 
Method* Student Achievement Implementation Plan Activities 
School G Benchmarks follow School established process | No external Conduct of parent, teacher, 
“Special Literacy | state guidelines for benchmarks, relating to evaluation; self- principal, and student 
Model”* student performance faculty participation, parent | assessment as part | surveys; review of 


participation and student if year-round SIP achievement scores; 
absences processes conduct of focus groups; 
reports from external 
technical assistance 
providers 


School H SIP describes No implementation No formal activities or |Evaluation of student 
Success for All benchmarks for benchmarks, other than plans performance based on 
student performance, | SFA implementation state assessment; 
as well as goals for monitoring implementation monitored 
attendance and by SFA; no written 
school ranking findings available 


School I No clear benchmarks in| No clear benchmarks in No clear evaluation School frequently reviews 
Lightspan place; did have goal place; did have goal of plan student achievement 

of increasing scores increased parent data; developer 

on state assessment participation previously used 
questionnaires and 
surveys to measure 
implementation; nor 
written evaluation 


School J Goals/benchmarks set |School set implementation External evaluator Schools reviewed and 
“Performance as part of SIP benchmarks annually (e.g., | focused on discussed student 
Tasks Model”* process teacher must prepare one standards, achievement data, 
performance task to be performance compared current data 
accepted by Performance measures and with data from earlier 
Task model database) alternative years, other schools and 
assessments (no district and state level 
formal plan) data 
School K Goals/benchmarks set |No implementation School did not have a |Schools reviewed and 


Success for All as part of SIP benchmarks clear evaluation discussed student 
process and covered plan achievement data, 
limited subjects compared current data 

with data from earlier 

years, other schools and 
district and state level 
data, and utilized SFA 
eight-week assessments 


School L School expected to Implementation benchmarks | District contracted End-of-year report 
“Behavioral reach benchmarks were stated in the CSRD with a consultant to contained district findings; 
Modification established by the application, and monitored conduct evaluation, final evaluation of CSRD 
Model”* district targeting by the external developer but turned delayed 


student performance responsibility over to 
(reading readiness, schools in 2001-02 
attendance, reading 

and math proficiency) 


*Locally developed methods are in quotes and have been given pseudonyms. 
(Continued on next page) 
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Exhibit 2-8 (Continued) 


Benchmarks 


School Name/ 

Method* Student Achievement 
School M District established 
Lightspan benchmarks for 

student performance, 
covering all grades 
and subjects 
School N SIP set benchmarks for 
Success for All performance, as well 
as goals around 
student attendance 
and school ranking 
School O District-mandated 
Success-in-the- benchmark tested as 
Making part of 
comprehensive 
reading program 
receiving greatest 
priority. 
School P Included extensive 
HSTW coverage of student 
performance on 
assessments and 
toward graduation 
requirements 
School Q District established 
Accelerated benchmarks based 
Schools on student scores on 
state assessment 
School R Three-year 
“Comprehensive benchmarks in SIP 
Reform Model”* for reading, writing, 


and mathematics 


Implementation 


No clear implementation 
benchmarks 


involvement and conflict 
resolution, but no 
information on how goals 
would be measured or 
benchmarks established 


Scale-up benchmarks had 
been met; method in use in 
all classrooms 


Listed as addressing 
organization and 
curriculum, and instruction 


Benchmarks in form of 
implementation rubric, 
including unity of purpose, 
empowerment, building on 
strengths, forging a vision, 
taking stock, setting 
priorities, establishing 
governance, embedding 
inquiry, and creating a 
collaborative culture 


Three-year benchmarks in 
SIP for use of technology 
and cooperation 


*Locally developed methods are in quotes and have been given pseudonyms. 
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Evaluation 


Plan 


Evaluation activities 
suggested by 
Lightspan (no 
written plan) 


School had goals for parental} No formal activities or 


plans 


District intends to 
conduct an impact 
assessment (using 
survey and student 
achievement data); 
no evidence of 
written plan 


State conducting 
formal evaluation of 
CSRD 
implementation 


No external evaluator; 
district provided 
assistance to 
schools in analyzing 
student 
achievement 


No written evaluation 
plan (except for plan 
for technology 
component, 
conducted as a 
state requirement) 


Activities 


Findings indicated that 
Lightspan students 
outperformed comparison 
students in the state 
assessment (report 
available after the 2001- 
02 SIP developed) 


Evaluation of student 
performance based on 
state assessment; 
implementation monitored 
by SFA 


School completed surveys; 
district analysis of data in 
progress; no written 
finding available 


Formal instrument used to 
evaluate instruction 
during classroom 
observations 


School reviewed student 
achievement data; cadres 
made recommendations 
for curricular or 
instructional reform 


No formal reporting of 
evaluation findings 


At the same time, the original CSRD language for the evaluation strategies 
(Component 8) does indeed address the implementation of reform, but the existing 
evaluation activities also have largely focused on assessing student performance (third and 
fourth columns in Exhibit 2-8.) [As a promising reform strategy, see “Clear Scope and 
Sequence and Quarterly Assessments Provide Frequent Measures of Student Growth,” in 
Appendix B.] 


2.3.3 High turnover among staff and students has resulted in transient levels of 
support within the school (Component 5) or parental and community involvement 
(Component 6). The field teams found it difficult to assess these two CSRD components 
because schools do not routinely monitor or maintain precise records regarding either type 
of support or involvement. In some cases, faculties did hold votes during the formative 
period when a research-based method was first being adopted. However, documentation 
for any subsequent faculty support was absent thereafter. Similarly, schools might have 
kept records of parent attendance at one or two key events during the year (e.g., the first 
open-school night), but they did not formally monitor or document other parental or 
community involvements. 


Despite the lack of records, the field teams made the following observations about the 
two components. First, high tumover among staff (e.g., Schools B, E, I, M, and R) or 
students (e.g., Schools C, E, L, and P) has meant that the extent of staff support or parental 
involvement can be transient: Any judgment regarding support or involvement therefore 
depends upon the timing of the data collection “snapshot.” Unfortunately, several of the 
schools had staff who were highly supportive of the reform effort but who then left the 
school (e.g., Schools C, E, M, and R); assessing staff support earlier would have shown 
strong internal support for the method and the reform effort, but such support likely 
dissipated greatly by the time of the site visit. At other schools, support among the staff 
increased over time as individual teachers became more familiar with the reform efforts. In 
a likewise manner, student turnover meant that the relevant body of parents also changed, 
and high involvement during the kickoff year of a reform effort may not have been 
sustained the following year. 


Second, strong principal leadership in initiating and supporting reform was a critical 
ingredient at many of the schools. [As a reform strategy, see “School Leadership Committed 
to Institutionalizing Reform Efforts,” in Appendix B.] However, turnover among principals 
also swayed support or involvement if, as is usually the case, the new principals had agendas 
and priorities different from their predecessors (e.g., Schools L, M, and R). 


Interestingly, any difficulties in assessing support and involvement also may be 
considered substantive findings, not just methodological barriers. Whereas time-limited 
innovations may call for staff support or parental involvement over shorter periods of time, 
comprehensive reform calls for broad support and involvement. The team found that 
schools had defined few if any strategies to produce such broad efforts. Other than a few 
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cases where either staff support for the reform effort (e.g., Schools A and F) or community 
involvement (e.g., Schools O and K) were strong and persistent, the observed support and 
involvement only involved isolated milestones (e.g., the original vote by a faculty to adopt 
a research-based method, the convening of a parents’ night, or the use of CSRD funds to 
hire a parent coordinator) but not sustained trends or patterns. [As promising reform 
strategy, see “Internal Collaborative Meetings Foster Inter-curricular Teaching and Teacher 
Buy-in to Schoolwide Issues,” and “Parents and Community Members Participate in 
School Governance and Goal Setting,” in Appendix B.] 


For other reasons related to the comprehensiveness of the reform effort, support or 
involvement also can be spotty. For instance, schools whose reforms were limited to single 
academic subjects would not necessarily have involved staff support or parental 
involvement outside of these subjects. Similarly, methods being scaled-up incrementally 
by grade level would not necessarily have called for universal support or involvement 
during the scale-up process. 


2.3.4 Most schools are in a position to coordinate or converge resources 
(Component 9), but resources for sustainability are still uncertain. The topics of 
coordination and sustainability (i.e., extending reform efforts beyond the period of CSRD 
funding) were joined because schools’ prospects for sustainability have been heavily driven 
by the need to identify new sources of funds.*’ Similarly, the resources to be coordinated 
under Component 9 have largely been considered budgetary resources by the schools. 


The significance of Component 9 also is related to the underlying spirit of reform. By 
definition, comprehensive reform should not be implemented as an “add-on” or incremental 
activity, but should affect all school operations and hence the entire set of a school’s 
resources—core building budget as well as funds from external sources not legally needing 
to be dedicated to a specific activity (e.g., Title I, Title H, Eisenhower, and other federal 
and state funds). Any actual and ongoing comprehensive reform would therefore logically 
require that the school’s resources be applied in a coordinated fashion. 


To collect data on these two topics of coordination and resources for sustainability, 
the field teams did focus on budget data but also collected other information, mainly from 
the principal, regarding the school’s funding intentions and priorities during the coming 
year(s). At 10 of the 18 schools, the field teams found evidence that both external and local 
funds were being coordinated in some manner to support the reform effort at the school. In 
most cases, the majority of that coordination involved Title I funding (see Exhibit 2-9). 


An initial observation is that coordination can be greatly facilitated if the planning for 


all of a school’s funds is represented within a single school improvement plan that 


*°Section 3 discusses sustainability more broadly. 
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Exhibit 2-9 


STATUS OF COORDINATION OF RESOURCES 
(Component 9) 
AND SUSTAINABILITY 


School Name/ 
Method* Types of Resource Coordination Evidence of Sustainability 
School A School already used Title |, other The principal was committed to providing the needed 
Accelerated external funds, and central budget— resources for the foreseeable future. ED and the 
Schools along with CSRD funds—to support state recognized the school for its reform efforts. The 
reform. Funds appeared to be used in | maintenance support needed by the research-based 
a coordinated manner in covering method is assumed to be modest. 
various reform activities. 
School B Nearly all CSRD funds were directed at |No funding streams appear to be available to continue 
High Schools that | professional development (with the school’s relationship with HSTW following the end of 
Work addition of some Title Il and other CSRD. The merger of this high school with a county 
funds). The coordinator position was school adds further uncertainty about sustainability of 
paid from the core budget. reform efforts, though the site visit team believes the 
merger will actually further reform efforts. 
School C The SIP embraces the use of funds from |The principal hopes to use Title | funds to continue 
Co-NECT all sources as part of an integrated after CSRD has ended. In addition, if the school can 
allocation plan. become a “demonstration” school, the developer of 
the research-based method also would then provide 
its own support. 
School D In 2001, district started requiring schools |The principal did not think the school would continue to 


Success for to have single plan to converge use SFA after CSRD funding ended for two reasons: 

All/Roots & Wings | resources, including CSRD funds, with | 1) a district move to site-based budgeting meant that 
common objectives (previously, each the school would have to identify funds to fill gap left 
funding source had its own plan). by CSRD (when large state cutbacks were looming), 
Convergence included external funding] and 2) the SFA reading curriculum was not submitted 
sources, but school reports no for state review, and, therefore, the school could not 
additional local funds available due to use State textbook funds to purchase SFA reading 
fiscal crisis. materials. 


School E A previous pattern of coordinating funds |The school has begun to be interested in adopting an 


Coalition of cannot be applied to the new charter entirely new research-based method, which may 
Essential Schools school. overlap with CES; most teachers show no interest in 
formally continuing with the CES method. 
School F School has been coordinating funds The school is searching for additional funds to replace 
Coalition of from multiple external and internal CSRD funding. However, main reform initiatives, 
Essential Schools sources to support reform effort in a some of which involved changes in school scheduling 
converging manner. and operations and not an outlay of funds, are likely 


continue independent of any funding success. 


School G CSRD funds were used almost entirely |Principal was confidant that new grant monies, along 
“Special Literacy for professional development, along with Title |, would continue to support future reform 
Model”* with Title | and other local funds. activities. 


*Locally developed methods are in quotes and have been given pseudonyms. 


(Continued on next page) 
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Exhibit 2-9 (Continued) 


School Name/ 


Method* 


School H 
Success for All 


School I 
Lightspan 


School J 
“Performance 
Tasks Model”* 


School K 
SFA 


School L 
“Behavior 
Modification 
Model”* 


School M 
Lightspan 


School N 
SFA 


Types of Resource Coordination Evidence of Sustainability 


The school decided to transition to the district 
curriculum and end support for the external SFA 
developer beginning in the 2001-02 school year. 

and district coordinate school’s funds in} Nevertheless, teacher indicated that they intended to 

relation to the school improvement continue using SFA-related classroom management 

plan. techniques and questioning strategies. 


Convergence of resources from many 
different external sources appeared to 
have taken place because the school 


School has funds from many external 
sources and prides itself in its 
entrepreneurship. However, multiple 
sources may have been more often 
associated with fragmentation rather 
than coordination of efforts. 


Although School |’s reform efforts could be described 
as unfocused, teachers and administrators expressed 
hope in Spring 2002 that things were beginning to 
improve. The 1st grade teachers were 
enthusiastically using Lightspan in the absence of 
external support. 


Some local funds were used in concert 
with CSRD to support the 


The school believes it can support the reform effort 
with Title | funds. However, the assignment of a new 
principal to the school may affect that plan; the 
associate principal being considered for the position 
stated he would support a modified use of the 
method, with only some components remaining. 


implementation of the research-based 
method. 


School has used multiple sources, 
including Title | and state grants, to 
support reform. Degree of 
convergence is unclear. 


Title | funds are expected to be used to continue SFA 
support after CSRD funding ends. However, 
continuation also depends on improved student 
performance, as well as SFA’s response to teacher’s 
concerns. 


In 2001, district started requiring schools |Funds to replace CSRD were still being sought, but a 
to have single plan to converge new administration and waning staff support 
resources, including CSRD, with appeared to be bigger threats to sustainability. By 
common objectives. Process has Spring 2002, teacher implementation of some 
brought all dollars into a single reform components of the method were already beginning to 
effort. However, school reports no lapse. 
additional local funds available due to 
fiscal crisis. 


‘A former principal appeared to have 
converged resources around 
Lightspan, its coordinator, computer 
equipment, and the other resources 
needed for implementation; by 2001- 


The school’s investment in purchasing the components 
of the method (the PlayStations, software, and 
television monitors), is evidence that some aspect of 
the method will remain. By the end of the third year, 
however, the method was used sparingly in one 
grade level. The school also learned that it would be 
merged with another “underperforming” elementary 
school in 2002-03, adding to uncertainty about future 
use of Lightspan. 


Withdrawal of district support for the research-based 
method jeopardized sustainability of reform effort 
beyond CSRD funding. The principal was skeptical 
about whether the school would continue to 
implement CSRD without new funding, and the staff 
indicated that the school would not be implementing 
SFA in 2002-03, even if funding were available. 


02, no funds were used to support 
implementation. 


School and district coordinated Title | 
funds with CSRD funds as part of the 
school improvement plan. However, 
school has had few other discretionary 
sources. 


*Locally developed methods are in quotes and have been given pseudonyms. 


(Continued on next page) 
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Exhibit 2-9 (Continued) 


School Name/ 
Method* Types of Resource Coordination Evidence of Sustainability 
School O School successfully coordinated funds |Because reform had been in progress prior to CSRD, 
SIM from multiple sources (e.g., Title | and | school anticipated continuing reform after CSRD. 
Title Il, as well as local funding), in External costs associated with research-based 
supporting reform activities. method were likely to diminish, and Title | and other 
funds would continue to be used, but the school also 
was exploring the availability of new sources of 
funding. 
School P School converged federal and local School and district staff strongly support continuation 
HSTW funds, primarily for professional of the reform effort. Extent of need for funds to cover 
development activities related to the external costs associated with the research-based 
reform effort. method were unclear. School staff were also unclear 
how the merger with the city high school would affect 
reform efforts. 
School Q School had started using other external |Main threat to sustainability is continued high rate of 
Accelerated funds, in a converging manner to staff turnover as well as threat of principal turnover. 
Schools support variety of reform activities prior | The principal hoped that the formation of a leadership 
to CSRD funding, and such team would make the reform efforts less dependent 
convergence has continued along with | on any single individual and therefore serve as a 
CSRD funding. mechanism for sustaining reform. 
School R School had not used a large portion of its|Some isolated initiative would be sustained, nut not as 
“Comprehensive CSRD funds in its final year of funding | part of a overall reform design; greater attention was 


Reform Model”* and obtained an extension on the grant] placed on the district-required “small learning 
(and, consequently, had not converged | communities. 
or coordinated funds). 


*Locally developed methods are in quotes and have been given pseudonyms. 


converges on common goals and objectives (see the coordination column of Exhibit 2-9). 
Of the 10 schools exhibiting converging funding sources, 3 had such a condition (Schools 
C, G, and H). Even in the absence of such site-based management conditions, 7 other 
schools reported coordination or convergence of resources to the field teams (Schools A, B, 
F, I, K, O, and P), with 3 of them mentioning integration with the building’s budget. Thus 
a total of 10 of the 18 schools appear to have been in a position to coordinate or converge 
their resources. Another 5 schools (D, J, L, N, and Q) reported coordination of federal 
funding sources to support reform, but did not have the necessary access to local funds in 
order to redirect them (in the case of School D and L, the principals reported that they had 
the ability to access local funds, but severe budget cuts meant that after obligatory 
allocations—i.e., teacher salaries—no funds remained). (See “Move to Site-Based 
Management Allows School to Utilize Slack Resources for Reform Initiatives,” in 
Appendix B.) Reports of lack of coordination came from only 3 schools: School E, which 
had ceased implementation; School M, which had made the capital expenditures associated 
with Lightspan and was now implementing the method at no cost; and School R, which had 
not really implemented CSRD, as previously discussed. 
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At the same time, the field teams found that the ability to coordinate funds did not 
necessarily coincide with the extent of a school’s discretionary resources. Yet, the latter 
may be critical for sustainability (see the second column of Exhibit 2-9). 


One reason for the need for discretionary resources is that a major CSRD 
component—e.g., the use of a research-based method—usually requires an outlay of 
additional funds, for such items as: a) on-site assistance, b) release time for professional 
development, c) salary support for on-site coordinators, d) the purchase of new types of 
supplies, and in some cases, e) hardware and software purchases. The availability of the 
CSRD funds has served well the needs of the startup periods in implementing these 
methods. Although the extent of outlays for maintaining a method after startup are 
reported to diminish, nearly all of the schools are unclear about the amount or source of any 
new funds needed after the CSRD award has ended. Title I funds are mentioned frequently 
as one source of such funds, but whether they will be sufficient also is unclear. 


The likelihood of sustainability also will be affected by non-funding conditions— 
mainly the possible lack of support for reform on the part of the school staff or the threat of 
staff or principal turnover. Of the 18 schools, 5 have reported such jeopardy (Schools B, D, 
L, M, and Q), in addition to School H, which already stopped implementing the research- 
based method. Section 3.5 examines this issue in greater detail. 


2.4 SUMMARY 


Across the 9 CSRD components, schools have devoted their greatest implementation 
efforts to the research-based method (Component 1); professional development 
(Component 3)—usually linked with the method; and external technical assistance and 
support (Component 7)—also usually linked with the method. Schools have established 
clear student performance benchmarks (Component 4). All of the other components have 
been less well developed. 
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3. CSRD IMPLEMENTATION AND SUSTAINABILITY: 
THE INTEGRATED FINDINGS 


This section first discusses district and state influences on the CSRD schools. The 
section then provides a summary assessment of CSRD implementation, using the district 
and state influences, along with other conditions, to identify three potentially different 
pathways to implementation. Finally, the section discusses the prospects for CSRD 
sustainability at the 18 schools. 


3.1 DISTRICT AND STATE INFLUENCES ON CSRD 


Although CSRD focuses on school activities, district and state conditions can be 
extremely important to the administration of CSRD. The federal CSRD awards are to state 
education agencies, who in turn make subawards to districts on behalf of specific schools. 
As aresult, any number of CSRD processes and outcomes may be influenced by the district 
and state policies and practices. 


To examine district and state conditions, the field teams conducted interviews, 
collected documents, and reviewed other data from the 18 schools’ district and state 
agencies. For the districts, site visits were made (in addition, district officials were 
occasionally interviewed while present at the schools at the time of the school site visit). 
Information about the state agencies more frequently came from short telephone 
conversations with state officials, searches of the agencies’ Web sites, and copies of state 
documents and reports provided by the district offices. 


Exhibit 3-1 shows the various district and state conditions that appeared relevant to 
the CSRD activities in the 18 schools. The exhibit pairs the schools, with each pair being a 
part of the same district and state, reflecting the original sample design. For each pair, the 
exhibit then enumerates salient conditions that might affect the work of CSRD, dividing the 
salient conditions into two categories: assistance or support in administering CSRD; and 
other policies or conditions potentially affecting CSRD implementation though not a direct 
part of administering CSRD. Each category is discussed in turn. 


3.1.1 Administering CSRD: Districts and states provide varying degrees of support. 
Both states and districts have explicit responsibilities in administering the funds for CSRD. 
However, beyond these core responsibilities, both districts and states varied widely in 
providing other forms of CSRD support or assistance. At one extreme (e.g., Schools M and 
R), the state education officials engaged schools in the CSRD application process; made 
site visits to monitor the initial progress of the schools; and convened annual meetings of 
the CSRD schools in the state, to facilitate the exchange of information on 
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Exhibit 3-1 


SCHOOL, DISTRICT, AND STATE CONDITIONS 


Schools State Conditions District Conditions 

School A Non-CSRbD related: Non-CSRD related: 

Accelerated ! State sets benchmarks for ! The district essentially only provides 

Schools improvement based on state assistance on an as-needed basis. 
assessment. Schools that fail to Professional development funds are 

and make progress, over time, may be distributed to schools; schools decide how 
closed. School E has been in lowest to use them. Other assistance only 
category and School A has been “in provided when requested by the schools. 

School E need of improvement,” in the state |! District sets benchmarks for improvement on 

Coalition of system. state assessments. 


Essential Schools ! District has vacillated on future of the School 


E, now has decided to continue the school 
as a charter school. 


School B Non-CSRD related: CSRD related: 


HSTW ! No state assessment has yet appeared]! District has provided staff support and 
related to reform. professional development in relation CSRD. 
and Non-CSRD related: 
! The district’s superintendent resigned effective 
School P the end of 2000-01. 
HSTW ! Board votes to merge Schools B and P intoa 
single high school, starting in 2003-04. 
School C Non-CSRD related: CSRD related: 
Co-NECT ! Performance on state assessment is !_ The district provided workshops in plan 
used to set goals. The state also development to ensure alignment. 
and provides a rubric for the Non-CSRbD related: 
development of district plans that is |! The district is suffering the loss of teachers due 
School G tied to state standards, requiring — to low teacher salaries and teacher 
a . . school plans to link funded activities shortages. Nearby districts and states are 
eperial Literacy with goals, benchmarks, and offering bonuses to attract teachers. 
Model” * strategies. ! The district has had a serious fiscal shortfall. 
School D CSRD related: CSRD related: 
Success for All ! Because of state timing in notifying !_ District decided which schools should apply; 
Roots & Wings schools of eligibility and application hired consultant to write CSRD proposals 
deadline, applications were written for the schools. 
Ane over the summer. At School L, Non-CSRD related: 
many staff were not present to ! District's strategic plan adopted under new 
participate in selection of the CSRD superintendent in 1998, calls for a new 
School L method. emphasis on high standards of academic 
Behavior Non-CSRD related: performance and has been a driving force 
Modification !_ State has put into place new high behind school reform efforts. 
Model” * stakes examinations. 


! State revising assessments to make 
them standards-based. 


*Locally developed methods are in quotes and have been given pseudonyms. 
(Continued on next page) 
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Exhibit 3-1 (Continued) 


Schools 


School F 
Coalition of 


Essential Schools 


and 
School O 


Success-in-the- 
Making (SIM) 


School H 
Success for All 


and 


School N 
Success for All 


School I 
Lightspan 


and 


School Q 
Accelerated 
Schools 


School J 
“Performance 
Tasks Model”* 


and 


School K 
Success for All 


District Conditions 


CSRD related: 
! District office provides extensive support to 
schools at the start of CSRD implementation 


State Conditions 


CSRD related: 

! State conducted monitoring visits to 
each site during the past year, and 
provided assistance. and conducts a mid-year checkup. 

Non-CSRD related: Non-CSRD related: 

! New state accountability system holds |! School O originally intended to implement Co- 
teachers and schools accountable NECT, but was “encouraged” by the district 
for student progress. School O to expand existing SIM method (after 
received an “F” rating which hurt its receiving an “F” rating). 
magnet program (and which they ! District implementing a reading program that 
believed was unfair). will phase out SFA. 


CSRD related: CSRD related: 

! The curriculum match between the ! The previous superintendent strongly 
CSRD method and the state supported the use of external methods for 
assessment is not perfect, causing reform and provided local funds to pay for 
School N to add a reading program them. 
specifically on skills addressed by |! Anew superintendent has ceased that 
the state assessment. practice, and only schools with CSRD funds 

Non-CSRD related: currently have external methods. Schools H 

! The state assessment and state stopped using its external method in 2001- 
accountability system is driving 02, and School N may have difficulty finding 
instruction in the schools. budget support for external methods after 
CSRD; other schools have adopted a local 
reading model aligned with the state 
assessment. 

Non-CSRD related: 

! The district also has started interim and high- 
stakes assessments, aligned with state 
standards. 


CSRD-related: 
The district identified schools that were ready 
for CSRD and notified only those schools 
providing assistance. about the availability of CSRD funding. 

Non-CSRbD related: Non-CSRD related: 

! Simultaneous with the CSRD award, |! District has helped schools prepare for state 
state budget cuts resulted ina assessment and to use data to improve 
several million dollar cut in district instruction. 
funding, in part resulting in the loss |! District has now gone two years with no 
of the district's P.D. department and professional development department. 
cuts to program evaluation. 


Non-CSRD related: CSRD related: 

! State designated School K as ! CSRD is coordinated by the district's director 
“needing improvement,” requiring it of federal and state programs, the assistant 
to select a research-based method superintendent for instruction, and the 
using CSRD funds. The state also director of research and evaluation. 
designated School J as a “priority }|Non-CSRD related: 
school,” receiving extra funds to ! District hired a new superintendent in 2000- 
support improvement efforts and 01. Superintendent supports reform but is 
being assigned a “critical friend.” focused on reorganizing the district’s central 

! The state assessment is standards Office. 
based. 


CSRD related: 
! State education staff conduct site ! 


visits to CSRD schools, also 


*Locally developed methods are in quotes and have been given pseudonyms. 
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Exhibit 3-1 (Continued) 


Schools State Conditions District Conditions 
School M CSRD related: CSRD related: 
Lightspan ! State convenes CSRD staffs annually |}! CSRD administered by Title | coordinator. 
to reinforce and support schools’ ! District helped School R develop its CSRD 
and efforts. application. 


! State makes site visits to CSRD Non-CSRbD related: 
S schools, also providing assistance. |! District assessments are standards-based and 
chool R ; : ; 
“Comprehensive Non-CSRD related: aligned with state. a 
»% |! School M was in state’s lowest ! However, School M received district quality 
Reform Model category, reducing staff morale, and performance award in same year as state 
esteem. placed it in lowest underperforming 
! State’s restructuring of school finance, category. 
and budget cutbacks, disrupt district 
and schools. 


*Locally developed methods are in quotes and have been given pseudonyms. 


CSRD experiences. In a similar manner (e.g., School L), a district completed the 
applications and selected the research-based method on behalf of the school. At the other 
extreme (e.g., School E) the state played little or no role beyond making the CSRD awards, 
and the district also offered little assistance to these schools in administering CSRD 
(School A). Exhibit 3-1 shows how the other states or districts assumed different postures 
in the CSRD administrative process. 


Among all the variants, an intermediate state and district posture would seem to be the 
best in serving comprehensive school reform. Such reform requires strong internal staff 
support within the school, but such reform is aimed at improving student achievement— 
which is assessed through district or state instruments (or both). In the most active role, if a 
district selects the candidate schools to be in the CSRD program, writes their proposals, and 
defines the research-based methods to be adopted, or if the state requires the school to 
participate in CSRD because of low achievement scores, the school may then have a poor 
understanding of its CSRD mission and may not become sufficiently empowered to 
undertake serious reform. In the most passive role, if districts and states leave schools 
entirely alone to deal with CSRD, the schools may receive insufficient guidance to 
understand the most desired curriculum and instructional practices relevant to the district or 
state’s student assessment instruments. The hypothesized intermediate role would be 
where districts and states: a) encourage and then support schools to make their own 
decisions about method selection and reform design—explicitly defining the alignment 
between these choices and the content of the district or state assessments, b) engage schools 
in developing their CSRD applications from their outset, and finally c) provide technical or 
other assistance to assure the continued alignment between the school’s reform initiative 
and the student outcome measures against which the success of the initiative will be 
judged. 


3.1.2 District and state policies not directly related to administering CSRD 
nevertheless can influence CSRD implementation. Exhibit 3-1 also enumerates many 


district and state policies not directly related to administering CSRD. These policies reflect 
ongoing district and state priorities independent of the CSRD program, and an important 
benefit of CSRD was its complementarity with these ongoing priorities. 


The original field protocol did not consider most of these conditions external to the 
administration of CSRD to be relevant topics of data collection. However, the field teams 
quickly encountered a host of external events that appeared to affect CSRD implementation 
directly. The conditions found included: 1) extremely limited professional development 
resources (see Schools I and Q); 2) the direct alignment of schools’ CSRD designs with 
district plans and state standards (Schools C and G); 3) districts reducing financial support 
for all external methods (Schools H and N); 4) decisions to allow a CSRD school to 
become a charter school (School E); 5) decisions to merge two CSRD schools (Schools B 
and P). Many states and districts also were suffering severe revenue shortfalls during this 
period of time, creating an uncertain if not disruptive environment for school reform. 


These non-CSRD-related district and state policies were found to be sufficiently 
important that Exhibit 3-1 divides the schools into two categories—one in which there 
appeared to be strong “vertical” influence (from state to district to school) and the other in 
which such influence appeared to be neutral. Strong influence could lead to either 
complementarity or conflict with CSRD. On the complementary side, CSRD provided 
Schools J and K, for instance, with resources and a compatible reform agenda to respond to 
their designation as underperforming schools in the state’s accountability system. On the 
conflicting side, the shifting content of state assessments could lead districts to use 
resources for alternative curricula and professional development that were contrary to those 
involved in the schools’ original CSRD plans or implementation (e.g., Schools H, N, and 
O). Likewise, a report suggesting that schools with block schedules performed more 
poorly on state assessments led another CSRD school (School B) to reconsider its ongoing 
implementation of a research-based method requiring the use of such schedules. 


In general, these external state and district conditions appear to be highly relevant to 
CSRD implementation. In fact, any tendency toward a “vertical” alignment among state, 
district, and school policies and practices may not only impact CSRD implementation but 
may very well be critical to the long-run sustainability of CSRD, discussed later in this 
section. 


3.2 SCHOOLS IMPLEMENTING COMPREHENSIVE SCHOOL REFORM 

Given the preceding discussion of district and state conditions, the analysis now turns 
to a summary assessment of the extent to which the 18 schools have been implementing 
CSRD. In this assessment, the more that a school was implementing the nine CSRD 


components, the greater was judged its strength of CSRD implementation. 
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Implementation Instrument and Measurement. Assessing the strength of 
implementation of a complex school intervention, such as comprehensive school reform, 
has neither a standard methodology nor a standardized instrument. As a result, the Field- 
Focused Study developed and used its own assessment instrument, shown in Appendix E. 
This assessment instrument is but the start of a tool with which to measure reform and 
requires much further work to formalize. Nevertheless, the instrument should provide a 
solid framework upon which others can improve. 


The instrument covers the 9 components, but deliberately weights the components 
differently. The weights for the different components reflected the Field-Focused Study 
team’s view of the importance of the 9 components, based on observations during the early 
rounds of site visits. The research-based method (9 points) and the comprehensiveness of 
design (11 points) have the highest proportion of the total points; professional development 
emerged as the third most impactful component; and the other components followed. This 
weighting system should be refined during future evaluations. 


The instrument asked the site visit teams to use their case reports and provide 
“objective” information for somewhat less than half of the items and to make “subjective” 
ratings for the remainder of the items.°° The teams had sought accurate and precise data on 
objective items, but such data were not always available or retrievable, and in many cases 
the “objective” response was based on the team’s best estimate. The scores for each school 
were tallied, and the results are shown in Exhibit 3-2. 


The 47-point instrument’s strength is that is operationalizes CSRD’s nine 
components. With no standardized instrument available to measure CSRD, the instrument 
allowed the evaluation team to break global concepts of reform into smaller operational 
components. If a school successfully implemented all aspects of all 9 components, 
therefore, it could achieve a maximum score of 47 points (thus the instrument has been 
referred to as the “47-point instrument”). The greatest threat to measuring reform in this 
manner is that it risks losing site of the “whole picture.” As a result, the final tallies were 
checked against a global judgment by the teams. 


A weakness of the instrument is that the Field-Focused Study did not have sufficient 
time or resources to establish the validity or reliability of the instrument to a level that 


°° For objective information, examples in the instrument included (see Appendix E): 2.1) the existence 
of a written design or plan for school reform; 2.2) the contents of the plan; 3.2) the range of professional 
development days required or taken by the average teacher per year; and 4.1) the number of academic 
subjects covered by measurable goals and benchmarks. For subjective information, examples of items 
included (also see Appendix E): 1.1) the implementation score for the research-based component; 3.1) 
whether professional development had a strong content focus; 5.3) interviewees voicing strong support or 
enthusiasm for reform; and 6.2) the level of parental involvement. 
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Exhibit 3-2 


SCORES FOR CSRD IMPLEMENTATION STRENGTH AT 18 SCHOOLS, 
USING 47-POINT INSTRUMENT* 


rate Te[e[*[r[e[*|'[>[«[1[™[slol[ lela 


[sectassroonsusng +a tasfsta}olr[ifo][:/e]s|s[a]ilile[s]o 


2. Comprehensive Design (11 points) 


1. Research-Based Method (9 points) 


Written Design or Plan 1 1 1 1 1 1 1 1 1 1 1 1 1 al 
Pconensorren—SSCSCid ‘To | [7 [*|7[oe{[e|4|7{[s|7l2]4[e|e|s]o2_ 


3. Professional Development (7 points) 


Sacer 


Classroom-Based PD 1 
[algnmentofeo———S—~id at Pt fa [a fl 
4. Measurable Goals and Benchmarks (4 points) 
[Number of Academic subjects Covered | 2 [2 ][2]2]0]e]?]*|]2]2]1]*]2]?2|2]s [J 2] 
[numberorGradescoveres ta fata tatot1{:f31]/*[1{i{1]:]1|:]11:]:] 
5. Support within the School (4 points) 
FFomalvosonmetod dT 2 [3 [a] ]o]a]2]1]0]1]1]0]°]1|1] 1]: ] 0] 
[suppor exceeded 75percene fa fofafa}o{a]s]alfo]1,ijolo]i|i]1|i]o] 
Support Voiced for Support 1 0 1 1 1 1 1 1 al 1 0 0 1 0 1 1 1 0 
[no pasentvorea————SSC~i 2 Po fo fa totaf2foli[1{olololo|i]1]:] 0] 
* See Appendix E for a copy of the instrument 
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Exhibit 3-2 (Continued) 
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6. Parent and Community Involvement (4 points) 
[NewFoms orParentinvonemene [112 [0]a]0]i1[]1]1]2]1]i]1]1]1]1]0]a] 0) 
[tevelorimowement 2 fa fofayolfaj[a1ti{z}o}2,olz2lz2{e2lo]z]o| 
/arueastone Communiyors 12 1213]3]1]1]ol1]1]1]11111111:]:]:]|:] 
7. External Technical Support and Assistance (4 points) 
omer external suppon 1 2it3a{f3atajtols[1[1]1]1}o,o,o{1]ol:]o]o] 


8. Evaluation Strategies (2 points) 


9. Coordination of Resources (2 points) 


[— coortinaton ofetemalfunds [a ]a]2]2]°][i]i]1]2|o]1]2]0].] 3] 
inati 1 


Coordination of External and Local Funds 


Pa fa [a fo 
P—Total(animum = alo [775 [a7 | © | 
ra 


would satisfy psychometric standards. The needed tool would be a “policy” tool, because 
of the instability of the policy environment—e.g., the 9 CSRD components became 11 
components a few years later under P.L. 110-103. Psychometric tools require a longer 
“shelf-life,” because the time and effort to establish their validity and reliability (and their 
generalizability) assumes that a set of substantive priorities will remain stable over a much 
longer period of time. As a result, “policy” tools seldom have sufficient evidence to 
establish their validity or reliability to a degree that will satisfy psychometric standards. 


The team did corroborate its use of the instrument, however, through an independent 
review of the cases by an analyst “blind” to the rating generated by the instrument (see 
Appendix F).*” Exhibit 3-3 compares the analyst’s rankings of the 18 schools with the 
rankings from the scores on the 47-point instrument. The results show a strong correlation 
between the two sets of rankings, with the rankings differing to a modest degree for a few 
schools (Schools O, Q, and L), but with the overall pattern achieving statistical significance 
(correlation coefficient = .734, n = 18, p < .001). 


Implementation: A solid majority of the schools implemented CSRD. Given 
support by the corroboratory procedure, the scores in Exhibit 3-2 were accepted as 
assessing the strength of CSRD implementation at the 18 schools. Nine schools (Schools 
A, B, D, F, G, J, K, O, and P) garnered 80 percent of the 47 points, with scores higher than 
37.6 points. These schools were labeled as “nearly fully” implementing CSRD. Another 
six schools (Schools C, H, I, L, N, and Q) garnered 65 percent of the 47 points, with scores 
between 30.6 and 37.6. These schools were labeled “partially” implementing CSRD. The 
remaining three schools were judged to be “poorly” implementing CSRD.”*® 


Other observers may differ with regard to the cutoff points chosen for these labels, or 
even the use of the labels themselves. However, the Field-Focused Study interpreted the 
overall set of scores as indicating that a solid majority of the 18 schools were implementing 
CSRD. As a result, student achievement scores at these schools could be expected to be 
affected, assuming some relationship between comprehensive school reform and student 
performance. 


37 An analyst independently read all of the case reports and ranked the 18 schools according to their 
level of implementation. In arriving at these judgments, the analyst deliberately avoided tallying the nine 
components, instead assuming a holistic perspective. The analyst was “blind” to the ratings that had been 
made by using the 47-point instrument. The analyst’s findings, together with the rationale for the holistic 
judgments, is found in Appendix F. 


8The 9-6-3 breakdown happens to be the same as that found for implementing the research-based 
method (see Sect. 2.1.1). However, the specific schools in each grouping were not synonymous and the 
measure for CSRD implementation is not synonymous with the implementation of the research-based 
method. 
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Ranking Based on Holistic Asse ssment 


in Appendix F 


Exhibit 3-3 


COMPARISON OF RANKINGS FOR 18 SCHOOLS, 
BASED ON TWO INDEPENDENT ASSESSMENTS OF 
STRENGTH OF CSRD IMPLEMENTATION 
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3.3 THREE EXAMPLES OF COMPREHENSIVE SCHOOL REFORM 
UNDER THE CSRD PROGRAM 


So far, the analysis has examined the 9 CSRD components, pertinent district and state 
conditions, and the overall extent of implementation based on progress on the 9 
components. The analysis has purposely been fragmented, to assure readers that the 9 
components and external conditions have been systematically examined. However, how all 
of these components and conditions might work together and become integrated within a 
specific school may not be readily apparent. 


To gain a more coherent picture, three schools were selected because they seem to be 
making good progress in implementing CSRD but appear to show a slightly different 
pattern of events in doing so. Appendix D contains brief summaries for each of the 18 
schools, and the descriptions for the three schools have been extracted from these reports.*” 


First Example. The first school (School F) began implementing Coalition of 
Essential Schools prior to its CSRD award (Component 1). Under CSRD, the school has 
expanded its use of the method and also now uses it in combination with other reform 
methods, including High Schools That Work. Together, the methods cover the school’s 
entire range of curriculum and instructional practices. The school has developed and 
shared a comprehensive vision for reform (Component 2), including a written plan that is 
well understood and supported by the entire staff (Component 5). The school is making 
progress on all CSRD components. For instance, large numbers of the faculty participate in 
learning groups and related professional development activities (Component 3), the school 
has successfully engaged parents and community members to serve on key committees 
(Component 6), and the school coordinates multiple external and internal resources in a 
converging manner (Component 9). However, the school’s clear goals and benchmarks 
(Component 4) are not matched by a clear evaluation plan or activity (Component 8). 


An apparent key to the first school’s progress has been the strong leadership of the 
principal and the low turnover of the teaching staff. The district also uses a performance 
and appraisal process that emphasizes professional development and is therefore 
compatible with the needs of comprehensive reform. 


Second Example. The second school (School P) only started its reform activities with 
its CSRD award and is implementing High Schools That Work as its main reform method 
(Component 1). The method’s comprehensive coverage provides a framework for all 
school improvement activities at the school (Component 2), and the staff and a core 


°° turn, the brief summaries are based on 18 in-depth and detailed cases reviewed by each school and 
available separately from COSMOS (see www.cosmoscorp.com). 
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leadership team fully support the research-based method and the reform plan (Component 
5). The school also is making progress on all CSRD components including the 
coordination of resources (Component 9). For instance, the state is conducting an external 
evaluation covering all 9 components, and the school itself uses a formal instrument to 
observe and evaluate classroom instruction (Component 8). The school-to-work aspect of 
the research-based method attracts a good deal of business and community support 
(Component 6). 


In support of reform at this second school, the district provides substantial staff 
support for CSRD activities and has compensated for any CSRD funding shortfalls in 
professional development. With strong external support from the developer of HSTW 
(Component 7), School P is now engaged in implementing a complementary research- 
based method that aligns its work with that of middle school feeders. 


Third Example. The third school (School G) also began its reform activities with the 
first year of CSRD funding, adopting a locally developed Special Literacy Model 
(Component 1). Although the method mainly pertains to grades K-3 and reading, the 
school’s reform activities also have embraced the other grades (4-6) and academic subjects, 
as driven by a comprehensive and converging school improvement plan (Component 2). 
The district requires the plan to be aligned with its own strategic goals and plan. Both must 
reflect, according to a state mandate, the state’s standards and assessment tools (the state 
also mandates that every school have an aligned plan). School G has therefore made 
progress on all CSRD components. For instance, the school’s staff strongly support the 
reform (Component 5), and the school closely monitors its progress on relevant goals and 
benchmarks (Component 4) that is key to the year-round school improvement planning 
process. School G’s small size, collaborative management style, and history of reform 
have all contributed strongly to the CSRD efforts. 


3.4 THREE PATHWAYS FOR IMPLEMENTING 
COMPREHENSIVE SCHOOL REFORM 


The essence of CSRD is its focus on improving student performance through 
comprehensive school reform. The three schools just described are among several that 
appear to be making solid progress along these lines but also may provide insight into the 
conditions under which schools are successful in implementing*’ schoolwide reform. 


4° The Field-Focused Study deliberately sought to identify implementation conditions substantively 
related to comprehensive school reform. The study did not attempt to examine more general implementation 
conditions, such as principal leadership, sufficiency of resources, and absence of turnover or other disruptive 


3-12 


Simply put, the three schools may represent three different sets of conditions—each 
characterized as a different pathway to comprehensive school reform. 


The first pathway, represented by School F, is implicit in the formal design of the 
CSRD program and is thus a component-driven pathway: A school envisions and 
implements comprehensiveness in terms of the 9 CSRD components, with many of the 
components compatible with the principles espoused by a “process”-oriented research- 
based method (Coalition of Essential Schools). The second pathway, represented by 
School P, is totally compatible with the formal design of the CSRD program but may be 
characterized as a method-driven pathway: A school implements a broad and 
comprehensive research-based method that affects virtually all school operations, including 
curriculum and instruction, and whose successful implementation substitutes for the need 
for any independent articulation of the 9 CSRD components.*' The third, represented by 
School G and also fully compatible with the formal design of the CSRD program, is a 
vertical-driven pathway: State or district strategic planning drives schools toward the nine 
components, including the research-based methods or strategies (Component 1), without 
specifically enumerating them. 


Whether the latter two pathways should be given as much weight as the first pathway, 
which best suits the language of the legislation, is unclear. Furthermore, no pathway is 
necessarily more immune than the others to such disruptive conditions as: high principal 
turnover rates, limited professional development resources, or planned or unplanned school 
restructuring. And, as previously described under state and district conditions, compared to 
the first two pathways, the third pathway does not necessarily make reform any more 
immune from shifts in state or district priorities, either. 


The potential insight presented by identifying these three pathways may be magnified, 
not by comparing the three pathways, but by hypothesizing the conditions when all three 
may be absent. A few of the 18 schools did not attend explicitly to the nine CSRD 
components, did not adopt a comprehensive method, and were not driven to strategic and 
performance-related planning by state or district policies. These schools therefore pursued 
none of the three pathways, and implementation of reform lagged or was even absent. 


effects. These general conditions remain important, but highlighting them would not have contributed to 
understanding CSRD, which was taken as the main objective of the current study. 


“'The distinction between the importance of method vs. program has continued to evolve over the 
CSRD program period. This study was designed to address the first two cohorts of CSRD schools, when such 
a distinction was not as evident. 
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3.5 SUSTAINING COMPREHENSIVE SCHOOL REFORM 


The experiences of the 18 schools in the Field-Focused Study also provide some clues 
regarding the sustainability of reform beyond the period of CSRD funding. To be recalled 
is that the CSRD program consists of three-year awards, of modest size, and are intended to 
serve (only) as a catalytic force in stimulating comprehensive reform at a school. Using 
these awards to implement short-lived changes—i.e., as in supporting a special “project”— 
does not fulfill the basic CSRD mission. Rather, schools must demonstrate a more 
sustained set of changes. However, neither the original legislation nor ED define the exact 
nature of the changes to be associated with sustaining a comprehensively reforming school. 


One implicit view, based mainly on the experiences of the New American Schools 
initiative, holds that the central changes to be sustained should be the practices associated 
with the research-based method supported by CSRD (Component 1). An alternative view 
is that comprehensive reform, though embracing a research-based method, also transcends 
it. Successful sustainability would therefore not be associated with the use of any 
particular method or combination of methods. A successfully reforming school may 
implement one or a combination of methods for a period of time and then later transition to 
other methods—but the transitions should reflect a progression toward continued school 
and student improvement rather than the “churning” of innovative practices. These two 
views led to two different interpretations of sustainability. 


To succeed according to the first view, a school must identify a continuing source of 
funds for a specific method, because the use of the method usually requires some annual 
expenditure, whether: a) to support the work of the external developer, or b) to provide 
professional development opportunities for training new staff and refreshing existing staff 
about the method. Although external developers reduce these annual costs as a school 
gains experience with their method over time, none of the 18 schools had unequivocally 
identified such ongoing sources of support for these activities. Title I funds were the most 
common candidate, but most schools could not be confident about making discretionary 
commitments in a context dominated by district or state budget cutbacks. At the same 
time, 10 of the 18 schools (Schools A, C, F, G, I, J, K, O, P, and Q) did indicate their 
intention to seek such support from external developers for the coming school year (2002- 
03).** Most of these schools, however, were not likely to support the method (or the 


“One of these schools (School K) may only have been using the CSRD-supported method because of a 
state requirement to adopt the method—because the school had been rated as a school “in need of 
improvement.” Whether the school will continue using the method if the school sufficiently improves and the 
state requirements are lifted remains to be seen. Alternatively, if the school fails to improve sufficiently, 
other sanctions may be applied and the continued use of the method also would be unclear. 
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needed professional development) at the same level as under CSRD, so the quality of the 
continued method may be diluted.*° 


To succeed according to the second view, a school substituting for its original CSRD- 
supported method should evidence a clear rationale related to school or student 
performance. Such was the case with four schools (Schools B, D, H, and N). For example, 
the faculty at one school (School B) that was otherwise doing well in implementing the 
CSRD-supported method decided not to proceed to the next phase, which required adoption 
of a block schedule—because of reports that schools with block schedules had done less 
well on state assessments than had schools without block schedules. A second school 
(School H) had already started using a district-developed curriculum aligned with the state 
assessment,“ instead of the original CSRD method. At the other two schools (Schools D 
and N), the districts had developed their own curricula that were believed to be more 
aligned with state assessments than the CSRD-supported method. Either school was free to 
continue using the original method, but the districts’ resources were devoted to supporting 
their own curricula and related professional development, and a strong possibility is that the 
two schools will transition to the district curricula. 


Pursuing the second view a step further raises the question of whether any of the 
original 10 schools continuing the CSRD-supported method might have been doing so in 
the face of an emerging and contrary rationale. Three of the 10 schools (Schools I, J, and 
P) may be vulnerable to this jeopardy. Two (Schools I and J) were continuing despite the 
lack of centrality of their method to their districts’ priorities; continuation may depend on 
the identity of a new principal, yet to be appointed at either school. The third (School P) 
was due to be merged with another school in 2003-2004, raising uncertainties because the 
other school’s principal already has been announced as the principal of the merged entity.”° 

Exhibit 3-4 places the 18 schools into a sustainability framework that accommodates 
both views. The most lenient condition occurs by accepting both views. Under this 
condition, 14 of the 18 Field-Focused Study schools were exhibiting some level of 


‘“.0f the 10 schools, two schools (Schools I and O) were in CSRD Cohort 1 (whose award period was 
1998-99 to 2000-01), so they already had shown an ability to continue the use of the method in their first 
“post-CSRD” year. Of the other three schools in Cohort 1, two (Schools E and R) had been rated as being the 
lowest in implementing CSRD in the first place (making the issue of sustainability moot). The status of the 
one remaining school in the cohort (School N) is discussed elsewhere in the text. 


“4Whether district-developed curricula can be considered a “research-based” method was beyond the 
scope of the Field-Focused Study. The districts’ main claim is the alignment between the substance of the 
curricula and that of the state assessments. 


“The other school happens also to be one of the 18 in the Field-Focused Study (School B). 
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Exhibit 3-4 


SUSTAINABILITY OF CSRD AT THE 18 SCHOOLS 


E, L, M, R 
Both Neither 
Likely to Continue Likely to Sustain 
Research-Based Some Method in 
Method Supported Relation to 
by CSRD Student Performance 


sustainability by the time their CSRD awards were ending. Accepting only the second 
view reduces the number to 11, and accepting only the first view reduces it to 10. 


The main barriers to sustainability continue to be the availability of slack resources, 
especially in light of states’ and districts’ revenue shortfalls in recent years. To sustain a 
reforming process, even when existing resources have been sufficiently coordinated and 
targeted to reform, still requires discretionary funds to support such essential activities as: 
adequate professional development (including support for teacher substitutes), especially in 
situations of high teacher turnover; time for common planning periods or teachers’ work on 
school leadership teams; and support for external technical assistance. Though such needs 
can be served with modest levels of funds, serving the needs is still a discretionary activity 
that may have to be ignored if the core school operations are underbudgeted. 
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4. CONCLUSIONS FROM THE FIELD-FOCUSED STUDY 


The Field-Focused Study arrived at the following conclusions in relation to the main 
research questions posed at the outset of the evaluation. 


Overall Implementation of CSRD. Using a 47-point instrument, the Field-Focused 
Study determined that 9 of the 18 schools were nearly fully implementing CSRD and its 9 
components, another 6 were partially implementing CSRD, and the remaining 3 were 
judged to be poorly implementing CSRD. On the whole, CSRD implementation was 
therefore uneven. 


Three Conditions (or “Pathways”) for Successful Implementation, Including 
District and State Influences. The study identified three different sets of conditions, or 
“pathways,” that appeared to be associated with the successful implementation of CSRD. 
The first pathway is a component-driven pathway, whereby a school uses the 9 CSRD 
components to guide the development and implementation of a comprehensive reform. 
The second is a method-driven pathway, whereby the school adopts and implements a 
comprehensive research-based method that affects virtually all school operations and 
whose successful implementation substitutes for the need for any independent articulation 
of the 9 CSRD components. The third is a vertical-driven pathway, whereby a school 
articulates and pursues the needed comprehensive strategies as a result of state and district 
requirements involving the setting of standards, use of appropriate assessment tools, and 
required and aligned district- and school-based strategic planning and improvement plans 
to meet performance standards. 


No single pathway was considered the “best” or preferred pathway, and no pathway 
was necessarily more immune than the others to such disruptive conditions as: high 
principal turnover rates, limited professional development resources, or planned or 
unplanned school restructuring. 


Sustainability of Comprehensive School Reform. Neither the original legislation nor 
ED define the exact nature of a school’s changes to be associated with sustaining a 
comprehensively reforming school beyond the three-year CSRD award period. As a result, 
the Field-Focused Study examined two different views of sustainability and judged the 18 
schools according to both. 


The first implicit view, based mainly on the experiences of the New American 
Schools initiative, holds that the central changes to be sustained should be the practices 
associated with the originally-supported research-based method. An alternative view is that 
comprehensive reform, though embracing a research-based method, also transcends it. By 
this second view, successful sustainability would not necessarily be associated with the 
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continued use of any particular method but could involve transitions from one research- 
based method to another, over time. The transitions would have to reflect a progression 
toward continued school and student improvement rather than the “churning” of innovative 
practices. 


Using the most lenient benchmark and accepting either of these two views as a 
criterion for assessing sustainability, 14 of the 18 schools were exhibiting a promising level 
of sustainability by the time their CSRD awards were ending. Accepting the 
“comprehensive reform” view alone reduces the number to 11; accepting the “research- 
based method” view alone reduces the number to 10. 


The main barriers to sustainability continued to be the availability of slack resources, 
especially in light of states’ and districts’ revenue shortfalls in recent years. To sustain a 
reforming process, even when existing resources have been sufficiently coordinated and 
targeted to reform, still requires discretionary funds to support such essential activities as: 
adequate professional development (including support for teacher substitutes), especially in 
situations of high teacher turnover; time for common planning periods or teachers’ work on 
school leadership teams; and support for external technical assistance. Though such needs 
can be served with modest levels of funds, serving the needs is still a discretionary activity 
that may have to be ignored if the core school operations are underbudgeted. Thus, the 
sustainability of comprehensive school reform is still questionable, given the current fiscal 
climate. 
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